Add a warding processor #3

micaelbergeron · 2016-09-14T18:26:25Z

~~Please consider this as a draft~~

This processor may be used to track down warding events, such as
Counterwarding, Placement or Expiry. It uses the combat log events
correlated with entity life state tracking.

Sample log output:

-- Ward placed
{"time":13,"type":"obs","key":"[78, 80]","slot":0,"x":78,"y":80,"z":130,"entityleft":false,"ehandle":4195051}
{"time":11,"type":"sen","key":"[78, 80]","slot":0,"x":78,"y":80,"z":130,"entityleft":false,"ehandle":4850406}

-- Ward expired
{"time":251,"type":"sen_left","key":"[78, 80]","slot":0,"x":78,"y":80,"z":130,"entityleft":true,"ehandle":4850406}
{"time":373,"type":"obs_left","key":"[78, 80]","slot":0,"x":78,"y":80,"z":130,"entityleft":true,"ehandle":4195051}

-- Ward countered
{"time":22,"type":"obs_left","key":"[78, 82]","slot":0,"attackername":"npc_dota_hero_abaddon","x":78,"y":82,"z":130,"entityleft":true,"ehandle":14631244}
{"time":50,"type":"sen_left","key":"[78, 80]","slot":0,"attackername":"npc_dota_hero_abaddon","x":78,"y":80,"z":130,"entityleft":true,"ehandle":9208152}

I still need to test for controlled minions counterwarding.

Feel free to add your thoughts.

this processor may be used to track down warding events, such as Counterwarding, Placement or Expiry. It uses the combat log events correlated with entity life state tracking.

micaelbergeron · 2016-09-14T18:29:02Z

@spheenik please have a look at the processor.

howardchung · 2016-09-14T18:29:32Z

Did you consider submitting this directly to clarity?

micaelbergeron · 2016-09-14T18:36:31Z

Yes I did, but as opendota would be the first project to use this code I think it could be extracted down the road. I don't know how often you update the clarity's version, so I'd rather have this code here than in clarity-examples.

spheenik

I like a lot that finally there's someone using the SpawnsAndDeaths example to good use! ;)

I might be porting the SpawnsAndDeaths processor from clarity-examples to clarity, so that it's events would be available to everyone. Then your Wards-Processor could be simplified.

One thing that came to my mind: What happens if a ward dies of old age, and another one gets countered on the same tick, and the update of the lifestate for the one dying of old age comes first?

micaelbergeron · 2016-09-14T19:18:27Z

I might be porting the SpawnsAndDeaths processor from clarity-examples to clarity, so that it's events would be available to everyone. Then your Wards-Processor could be simplified.

I think instead of include the SpawnsAndDeaths, I think a OnFieldPathChanged(Class entityClass, FieldPath path) would be a more useful implementation. Just a suggestion.

One thing that came to my mind: What happens if a ward dies of old age, and another one gets countered on the same tick, and the update of the lifestate for the one dying of old age comes first?

Your question is very good, in fact I didn't yet found a way to handle these. I have a replay which represent this exact problem and the output is wrong. This is my next problem to tackle, but I could use your insights on this one.

--- 4 wards killed on the same tick, notice the order (sen, obs, sen, sen)
{"time":74,"type":"DOTA_COMBATLOG_DEATH","value":0,"attackername":"dota_unknown","targetname":"**npc_dota_sentry_wards**","sourcename":"dota_unknown","targetsourcename":"npc_dota_sentry_wards","attackerhero":false,"targethero":false,"attackerillusion":false,"targetillusion":false,"inflictor":"dota_unknown","gold_reason":0,"xp_reason":0}
{"time":74,"type":"DOTA_COMBATLOG_DEATH","value":0,"attackername":"dota_unknown","targetname":"**npc_dota_observer_wards**","sourcename":"dota_unknown","targetsourcename":"npc_dota_observer_wards","attackerhero":false,"targethero":false,"attackerillusion":false,"targetillusion":false,"inflictor":"dota_unknown","gold_reason":0,"xp_reason":0}
{"time":74,"type":"DOTA_COMBATLOG_DEATH","value":0,"attackername":"dota_unknown","targetname":"**npc_dota_sentry_wards**","sourcename":"dota_unknown","targetsourcename":"npc_dota_sentry_wards","attackerhero":false,"targethero":false,"attackerillusion":false,"targetillusion":false,"inflictor":"dota_unknown","gold_reason":0,"xp_reason":0}
{"time":74,"type":"DOTA_COMBATLOG_DEATH","value":0,"attackername":"dota_unknown","targetname":"**npc_dota_sentry_wards**","sourcename":"dota_unknown","targetsourcename":"npc_dota_sentry_wards","attackerhero":false,"targethero":false,"attackerillusion":false,"targetillusion":false,"inflictor":"dota_unknown","gold_reason":0,"xp_reason":0}

-- 4 triggered events (sen, sen, sen, obs)
{"time":74,"type":"**sen_left**","key":"[80, 80]","slot":0,"attackername":"dota_unknown","x":80,"y":80,"z":130,"entityleft":true,
"ehandle":2228562}
{"time":74,"type":"**sen_left**","key":"[80, 80]","slot":0,"attackername":"dota_unknown","x":80,"y":80,"z":130,"entityleft":true,
"ehandle":12026197}
{"time":74,"type":"**sen_left**","key":"[78, 82]","slot":0,"attackername":"dota_unknown","x":78,"y":82,"z":130,"entityleft":true,
"ehandle":13222238}
{"time":74,"type":"**obs_left**","key":"[76, 84]","slot":0,"attackername":"dota_unknown","x":76,"y":84,"z":130,"entityleft":true,
"ehandle":10502497}

To me that means the killer of the second sentry is wrong (and the obs too).

I'm wondering how frequent same-tick events happens though.
Thank you for your time.

spheenik · 2016-09-14T19:34:57Z

Hmm, the problem is that the field path differs between different entity types, and there is no way to parametrize the annotation to tell it which fieldpaths you are interested in. One could hack it by doing @OnFieldPathChanged(entityClassPattern, propertyName) and postpone the computation from propertyNameto fieldPath to when that can be done. You could then do

@OnFieldPathChanged("CDOTA_NPC_Observer_Ward(_Truesight)?", "m_hLifeState")
public void onLifeStateChanged(Matcher matcher, Entity e, FieldPath fieldPath) {
  // use matcher to find out if truesight or not, 
  // and get the current value from the entity using the fieldpath
}

howardchung · 2016-09-15T02:20:13Z

pom.xml

                    </execution>
                </executions>
            </plugin>
+            <!-- for some reason the shade-plugin doesn't work well with clarity processors.


preferably we should find a way to avoid needing to swap this dependency.

I took like 3 hours today trying to make this work. Maybe @spheenik can give some guidance because the shaded jar will crash upon populating the processors. It seems it cannot find the annotations or whatever.

IIRC this produces a jar with a different name. We have to make sure the output file is named the same, or update the Dockerfile to start the correct file.

The problem here is that there is an annotation scanner in clarity (org.atteo.classindex) that scans annotations on compile time, and produces a file META-INF/annotations/skadistats.clarity.event.Provides which contains a list of all classes with Provides-annotations. If you use shade, you got to make sure this file is present and correctly merged.

howardchung · 2016-09-15T02:21:36Z

src/main/java/yasp/Main.java

+    public static void main(String[] args) {
+	try { 
+
+	    if (args.length > 0 && args[0].equals("--file")) { System.exit(parse(args[1])); }


Fix brace spacing

To be honest, if you want to include the Main class, I'd rather have a proper arg-parsing, like Apache Commons CLI or similar.

The current code only has the server case. i assume you added these additional startup methods to make testing easier. Previously, I just started the server and then used curl to POST a replay to it. Up to you if you want to keep them around.

howardchung · 2016-09-15T02:22:20Z

src/main/java/yasp/Main.java

+	try { 
+
+	    if (args.length > 0 && args[0].equals("--file")) { System.exit(parse(args[1])); }
+            else if (args.length >0 && args[0].equals("--")) { System.exit(parseStream(System.in, System.out)); }


fix >0 spacing

howardchung · 2016-09-15T02:23:13Z

src/main/java/yasp/Main.java

+            else { startServer(args); }
+	}
+	catch (Exception e) {
+	    e.printStackTrace();


do you need to catch the exception here? If we have one, we should crash since the startup is incorrect.

howardchung · 2016-09-15T02:23:55Z

src/main/java/yasp/Main.java

    }

+    static int parse(String replay_file) throws Exception {


call this parseFile

howardchung · 2016-09-15T02:24:08Z

src/main/java/yasp/Main.java

    }

+    static int parse(String replay_file) throws Exception {


use camelCase

and call it fileName

howardchung · 2016-09-15T02:24:47Z

src/main/java/yasp/Main.java

-        server.createContext("/", new MyHandler());
-        server.setExecutor(java.util.concurrent.Executors.newFixedThreadPool(4));
-        server.start();
+    static int parseStream(InputStream is, OutputStream os) throws IOException {


I think methods should have visibility declared.

howardchung · 2016-09-15T02:25:22Z

src/main/java/yasp/Main.java

+	catch (Exception e)
+	{
+	    e.printStackTrace();
+            return -1;


extra space

howardchung · 2016-09-15T02:27:57Z

src/main/java/yasp/Parse.java

    @OnEntityLeft
-    public void onEntityLeft(Context ctx, Entity e) {
-        processEntity(ctx, e, true);
+    public void onEntityExistenceChanged(Context ctx, Entity e) {


Since the annotation here is still OnEntityLeft, I think the method name should reflect that.

Actually, if this is only here for cosmetics entities, you should probably just keep the OnEntityEntered and remove OnEntityLeft (assuming ward-related annotated methods will be called separately)

Yeah that could be a good way to catch this.

howardchung · 2016-09-15T02:28:22Z

src/main/java/yasp/Parse.java

@@ -459,6 +459,19 @@ public void onTickStart(Context ctx, boolean synthetic) {
            }
        }
    }
+
+    @OnWardCountered
+    public void onWardCountered(Context ctx, Entity e, String killer_hero_name) {


howardchung · 2016-09-15T02:29:20Z

src/main/java/yasp/Parse.java

+
+    @OnWardExpired
+    @OnWardPlaced
+    public void onWardExistenceChanged(Context ctx, Entity e) {


can you unify this method with the previous one and detect from context which case it is?

I don't think I can do that as the signatures are different.

howardchung · 2016-09-15T02:31:00Z

src/main/java/yasp/Parse.java

+	}
+    }
+
+    Entry buildWardEntry(Context ctx, Entity e) {


add method visibility

howardchung · 2016-09-15T02:31:45Z

src/main/java/yasp/Parse.java

+	entry.key = Arrays.toString(pos);
+	entry.ehandle = e.getHandle();
+
+	if (entry.entityleft)


use braces for all if statements

howardchung · 2016-09-15T02:33:11Z

src/main/java/yasp/processors/warding/Wards.java

+@Provides({ OnWardCountered.class, OnWardExpired.class, OnWardPlaced.class })
+public class Wards {
+
+    private static final Set<String> WARDS_DT_CLASSES = new HashSet<String>(Arrays.asList(


possibly add one for Source 1 entity names as well.

I think there's a shorter way to populate a set with elements that doesn't require Arrays.asList of a new String[].

Yes but in this case .contains is O(1).

yes, you'd keep using a hashset. Unfortunately it doesn't look like set literals are in Java 8: http://stackoverflow.com/questions/10705705/how-do-i-use-collection-literals-in-java-7

I tried to find the literal notation not to avail.

howardchung · 2016-09-15T02:35:08Z

src/main/java/yasp/processors/warding/Wards.java

+        }
+    } 
+
+    @Initializer(OnWardCountered.class)


consider changing name to OnWardKilled since that's a more precise description of what happened (a combat log event indicating a killed ward)

Seems legit.

howardchung · 2016-09-15T03:14:13Z

src/main/java/yasp/processors/warding/Wards.java

+
+    private static final Set<String> WARDS_TARGET_NAMES = new HashSet<String>(Arrays.asList(
+        new String[] {
+            "npc_dota_observer_wards",


are you sure it's supposed to be plural?

{"time":22,"type":"DOTA_COMBATLOG_DEATH","value":0,"attackername":"npc_dota_hero_abaddon","targetname":"npc_dota_observer_wards","sourcename":"npc_dota_hero_abaddon","targetsourcename":"npc_dota_observer_wards","attackerhero":true,"targethero":false,"attackerillusion":false,"targetillusion":false,"inflictor":"dota_unknown","gold_reason":0,"xp_reason":0}

Don't get me started on Valve's conventions.

this processor may be used to track down warding events, such as Counterwarding, Placement or Expiry. It uses the combat log events correlated with entity life state tracking.

upon a same-tick event, there is a chance that the ward kill will be credited to the wrong attacker, but this commit limits the impact of this caveat by separating the wards from the sentries. thus you will always be credited a ward kill if you last hit a ward

micaelbergeron · 2016-10-24T21:45:40Z

I've finally found time to finish this parser. I've tested on 5 replays with more than 10 counter-wards and I hit 100% success rate. I think this can be merged.

I'll start working on the UI.

howardchung · 2016-10-25T02:31:27Z

You can preview the wardmap here:
http://beta.opendota.com/matches/2731466093/vision

It currently doesn't support timeline (it just displays all the wards placed when the user is selected).

howardchung · 2016-10-25T02:32:41Z

Can you sync master to this PR?

howardchung · 2016-10-25T02:29:56Z

Dockerfile

@@ -12,4 +12,6 @@ WORKDIR /usr/src/parser
 ADD . /usr/src/parser
 RUN mvn -q -f /usr/src/parser/pom.xml clean install -U

+EXPOSE 5600


how did this work before? port option passed via Docker command line?

This is because the new docker-compose is using named services to access the other services instead of going through the host networking. It is good practice to EXPOSE listening ports on your docker components, it makes it easy to expose the service to other components.

As you can see in docker-compose.yml

environment: PARSER_HOST: http://odota-parser:5600 POSTGRES_URL: postgresql://postgres:postgres@odota-postgres/yasp POSTGRES_TEST_URL: postgresql://postgres:postgres@odota-postgres/yasp_test READONLY_POSTGRES_URL: postgresql://readonly:readonly@odota-postgres/yasp REDIS_URL: redis://odota-redis:6379/0 REDIS_TEST_URL: redis://odota-redis:6379/1 CASSANDRA_URL: cassandra://odota-cassandra/yasp CASSANDRA_TEST_URL: cassandra://odota-cassandra/yasp_test

I think docker-compose automatically expose the image EXPOSED ports.

howardchung · 2016-10-25T02:35:27Z

src/main/java/yasp/processors/warding/Wards.java

+        WARDS_TARGET_NAMES = Collections.unmodifiableSet(new HashSet<>(target_by_dtclass.values()));
+    }
+
+    // simple inderection map


indirection?

Opps, that's not relevant anymore.

howardchung · 2016-10-25T12:01:01Z

Dockerfile

@@ -12,4 +12,6 @@ WORKDIR /usr/src/parser
 ADD . /usr/src/parser
 RUN mvn -q -f /usr/src/parser/pom.xml clean install -U

+EXPOSE 5600
+
 CMD ["java", "-jar", "-Xmx256m", "/usr/src/parser/target/stats-0.1.0.jar", "5600"]


will you need to update this for the jar name?

No, I've changed the pom.xml to output the same filename as before.

howardchung · 2016-10-25T12:05:05Z

@spheenik could you give this another once-over?

micaelbergeron · 2016-10-25T14:13:54Z

@spheenik I'm not using the new stuff you've pushed to clarity 2.2, I'll wait for opendota to catch up the version before doing this.

howardchung · 2016-10-25T17:17:17Z

You can bump the version to 2.2 in this PR. I don't believe there are any breaking changes.

dartharaknus · 2016-12-16T01:46:54Z

src/main/java/yasp/Parse.java

+
+    private Entry buildWardEntry(Context ctx, Entity e) {
+	Entry entry = new Entry(time);
+        boolean isObserver = !e.getDtClass().getDtName().contains("TrueSight");


Extra space

howardchung · 2017-04-08T18:31:20Z

Needs to be rebased against master (and I think clarity 2.2 has some functionality that can make this PR simpler)

Add a warding processor

7e49bb5

this processor may be used to track down warding events, such as Counterwarding, Placement or Expiry. It uses the combat log events correlated with entity life state tracking.

spheenik approved these changes Sep 14, 2016

View reviewed changes

howardchung reviewed Sep 15, 2016

View reviewed changes

howardchung requested changes Sep 15, 2016

View reviewed changes

howardchung and others added 2 commits September 14, 2016 19:48

Update postbuild.sh

2dab48f

reword annotions and fix code style

c7be138

howardchung reviewed Sep 15, 2016

View reviewed changes

howardchung added 5 commits September 14, 2016 20:27

Update .travis.yml

74474b1

extract player_slot for cosmetics

a72fcaa

Merge branch 'master' of github.com:yasp-dota/parser

47530fb

add some owner hero entity data

510d6f2

update rebuild script

063e10c

spheenik mentioned this pull request Sep 28, 2016

How to distinguish the ward is destroyed by which hero? skadistats/clarity#109

Closed

micael.bergeron added 5 commits October 6, 2016 14:32

Add a warding processor

39c1428

this processor may be used to track down warding events, such as Counterwarding, Placement or Expiry. It uses the combat log events correlated with entity life state tracking.

reword annotions and fix code style

cb76494

Merge branch 'master' of github.com:micaelbergeron/parser

966684a

update the Dockerfile to expose the parser's port

55a0a45

howardchung reviewed Oct 25, 2016

View reviewed changes

remove the indirection map (was not used)

57fa27b

howardchung mentioned this pull request Oct 26, 2016

Display the player who counterwarded odota/core#1208

Closed

3 tasks

micaelbergeron mentioned this pull request Nov 11, 2016

Match vision tab overhaul odota/web#469

Merged

dartharaknus reviewed Dec 16, 2016

View reviewed changes

Argn0 mentioned this pull request Dec 8, 2017

cap ward lifespans and exclude some wards from average odota/web#1379

Merged

evaldobratti mentioned this pull request Oct 14, 2018

ward killer #24

Merged

Add a warding processor #3

Are you sure you want to change the base?

Add a warding processor #3

Conversation

micaelbergeron commented Sep 14, 2016 • edited Loading

micaelbergeron commented Sep 14, 2016

howardchung commented Sep 14, 2016

micaelbergeron commented Sep 14, 2016

spheenik left a comment

Choose a reason for hiding this comment

micaelbergeron commented Sep 14, 2016 • edited Loading

spheenik commented Sep 14, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micaelbergeron Sep 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

micaelbergeron Sep 15, 2016 • edited Loading

Choose a reason for hiding this comment

micaelbergeron commented Oct 24, 2016

howardchung commented Oct 25, 2016

howardchung commented Oct 25, 2016

Choose a reason for hiding this comment

micaelbergeron Oct 25, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

howardchung commented Oct 25, 2016

micaelbergeron commented Oct 25, 2016

howardchung commented Oct 25, 2016

Choose a reason for hiding this comment

howardchung commented Apr 8, 2017

micaelbergeron commented Sep 14, 2016 •

edited

Loading

micaelbergeron commented Sep 14, 2016 •

edited

Loading

spheenik commented Sep 14, 2016 •

edited

Loading

micaelbergeron Sep 15, 2016 •

edited

Loading

micaelbergeron Sep 15, 2016 •

edited

Loading

micaelbergeron Oct 25, 2016 •

edited

Loading