Seeing error in getDomain

[keith-turner]

Repeatedly seeing the following error in getDomain.

16:46:24.087 [pool-10-thread-38] WARN  io.fluo.core.worker.WorkTask - Failed to execute observer CollisionFreeMapObserver notification : um:u:4 fluoRecipes cfm:um  153397
java.lang.RuntimeException: java.text.ParseException: Invalid host: whattoexpect.co.au
        at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.getDomain(UriMap.java:148) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.updatingValues(UriMap.java:135) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.recipes.map.CollisionFreeMap.process(CollisionFreeMap.java:203) ~[fluo-recipes-core-1.0.0-beta-1-SNAPSHOT.jar:1.0.0-beta-1-SNAPSHOT]
        at io.fluo.recipes.map.CollisionFreeMapObserver.process(CollisionFreeMapObserver.java:44) ~[fluo-recipes-core-1.0.0-beta-1-SNAPSHOT.jar:1.0.0-beta-1-SNAPSHOT]
        at io.fluo.core.worker.WorkTask.run(WorkTask.java:69) ~[fluo-core-1.0.0-beta-2-SNAPSHOT.jar:1.0.0-beta-2-SNAPSHOT]
        at io.fluo.core.worker.NotificationProcessor$2.run(NotificationProcessor.java:131) [fluo-core-1.0.0-beta-2-SNAPSHOT.jar:1.0.0-beta-2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
Caused by: java.text.ParseException: Invalid host: whattoexpect.co.au
        at io.fluo.webindex.data.util.LinkUtil.createURL(LinkUtil.java:42) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.webindex.data.util.LinkUtil.getHost(LinkUtil.java:78) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.webindex.data.util.LinkUtil.hasIP(LinkUtil.java:87) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.webindex.data.util.LinkUtil.getReverseTopPrivate(LinkUtil.java:98) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        at io.fluo.webindex.data.fluo.UriMap$UriUpdateObserver.getDomain(UriMap.java:146) ~[webindex-data-0.0.1-SNAPSHOT.jar:na]
        ... 8 common frames omitted

[keith-turner]

This issue is caused by using different versions of guava in Spark vs Fluo. When the Spark job runs it uses guava 11 (because of yarn I think). With the older version of guava the domain is ok. When a Fluo Observer processes the data, a newer version of guava is in use. The newer version of guava no longer thinks the domain is ok.

[keith-turner]

I experimented with making my webindex observers use guava 11 (which is what spark is using). However, the Fluo framework uses guava 13 which makes it difficult for an application to use guava 11. Dependency isolation (apache/fluo#536) would be nice.

[keith-turner]

Found the following site. We may be able to use the java software plus list from that site. The nice thing there is that the list is not built into the software like it is in guava.

https://publicsuffix.org/

[tfmorris]

Java lib that uses public suffix list: https://github.com/hamano/regdom4j/