Help with testing Fred - Freelands MP Custom AI v0.14.15
Moderator: Forum Moderators
Help with testing Fred - Freelands MP Custom AI v0.14.15
Hello All,
Updated on 28 Feb 2020 with release of v0.14.15 for Wesnoth 1.14. Note that, as of v0.14.11, Wesnoth 1.12 is no longer supported.
As some of you might know, a few of us have been working on custom AIs on and off for several years now. One of the results of this work are the Micro_AIs that were introduced in Wesnoth 1.12. Another are a couple of experimental MP AIs that are part of the AI Modification Demos add-on, dubbed Ron and Fred. Ron is a (comparatively) simple rush AI that was introduced into mainline quite some time ago (he's called the 'Experimental AI' there), while Fred is much more complex and still a work in progress. If you want to read more about the background, there's a rather voluminous thread in the Coder's Corner forum.
A couple months ago I picked up work on Fred again after a year-long hiatus and I believe that he might now be ready for introduction to this forum and more general testing by (human) players. I do not have the time to do a lot of "manual testing", so mostly I watch games of Fred vs. the Default AI. While he does very well in those match-ups, we all know that that is not necessarily an indication that he will also be an interesting or fun opponent for a human player. Thus, I would like to solicit your help with testing of Fred.
Currently, Fred is only set up to play Northerners as Side 1 on the Freelands MP map. Extending this to Side 2 and other factions is relatively easy (and, in fact, mostly in place already), while playing on other maps is a very different story and might never happen. Or it might. We will have to see how he does on Freelands first...
More information on how to use Fred, what kind of testing/feedback would be useful, known problems etc. follows in the sections below. Read at your own risk, big walls of text ahead.
Any testing, feedback, comments etc. would be greatly appreciated.
Cheers and thanks,
mattsc & Fred
---------------------------------------------------
Status as of AI-demos v0.14.15: (this information will be updated as new versions are released)
Updated on 28 Feb 2020 with release of v0.14.15 for Wesnoth 1.14. Note that, as of v0.14.11, Wesnoth 1.12 is no longer supported.
As some of you might know, a few of us have been working on custom AIs on and off for several years now. One of the results of this work are the Micro_AIs that were introduced in Wesnoth 1.12. Another are a couple of experimental MP AIs that are part of the AI Modification Demos add-on, dubbed Ron and Fred. Ron is a (comparatively) simple rush AI that was introduced into mainline quite some time ago (he's called the 'Experimental AI' there), while Fred is much more complex and still a work in progress. If you want to read more about the background, there's a rather voluminous thread in the Coder's Corner forum.
A couple months ago I picked up work on Fred again after a year-long hiatus and I believe that he might now be ready for introduction to this forum and more general testing by (human) players. I do not have the time to do a lot of "manual testing", so mostly I watch games of Fred vs. the Default AI. While he does very well in those match-ups, we all know that that is not necessarily an indication that he will also be an interesting or fun opponent for a human player. Thus, I would like to solicit your help with testing of Fred.
Currently, Fred is only set up to play Northerners as Side 1 on the Freelands MP map. Extending this to Side 2 and other factions is relatively easy (and, in fact, mostly in place already), while playing on other maps is a very different story and might never happen. Or it might. We will have to see how he does on Freelands first...
More information on how to use Fred, what kind of testing/feedback would be useful, known problems etc. follows in the sections below. Read at your own risk, big walls of text ahead.
Any testing, feedback, comments etc. would be greatly appreciated.
Cheers and thanks,
mattsc & Fred
---------------------------------------------------
Status as of AI-demos v0.14.15: (this information will be updated as new versions are released)
How to use Fred
General remarks and goals
How can you help?
Known issues and future work
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help requested with testing Fred - Freelands MP Custom A
Here is a replay where I got a long evaluation on the last / second to last turn. This may be not surprising as it is the first time there are so many units in such close proximity during this match.
The only really clear mistake I noticed is that Fred prioritizes the retreat of ordinary units out of attack range a bit higher than using them to protect his leader when the keep is being surrounded. You see a bit of this as well in the replay; there are still units left to surround and protect their king but they are too afraid to do their duty.
The only really clear mistake I noticed is that Fred prioritizes the retreat of ordinary units out of attack range a bit higher than using them to protect his leader when the keep is being surrounded. You see a bit of this as well in the replay; there are still units left to surround and protect their king but they are too afraid to do their duty.
- Attachments
-
- slow_down_at_turn_18.gz
- (20.85 KiB) Downloaded 858 times
Re: Help requested with testing Fred - Freelands MP Custom A
I second that post.
I like how fred operates in teams and does not try to attack in vain. It is really different then the rushing default ai with regards to this.
It occurred to me that once the situation is evaluated to be in favor of the other player, fred switches to be overly defensive, thus giving up
initiative completely (however n=2).
This is outstanding work, and I will play it again sometime.
I like how fred operates in teams and does not try to attack in vain. It is really different then the rushing default ai with regards to this.
It occurred to me that once the situation is evaluated to be in favor of the other player, fred switches to be overly defensive, thus giving up
initiative completely (however n=2).
This is outstanding work, and I will play it again sometime.
- Attachments
-
- 2p_—_The_Freelands_replay_01.gz
- (29.69 KiB) Downloaded 824 times
-
- 2p_—_The_Freelands_replay.gz
- Felt pressured at first
- (30.54 KiB) Downloaded 827 times
Re: Help requested with testing Fred - Freelands MP Custom A
Cold Steel, benjam: Thank you! I really appreciate you taking the time to test Fred and writing up your comments!
The replays are very helpful and I must say that I am not disappointed by what I am seeing. I mean, clearly Fred does things that are not ideal. But then, some of those I'm still working on, and overall I am not going for an AI that doesn't make mistakes and can beat a human player on even terms anyway. As long as he provides a different kind of challenge than the default AI and doesn't screw up really badly too often, I'm happy.
Yeah, Fred's still pretty bad at this "endgame" thing (both when losing and when winning, actually). Your replays are going to be great test cases for trying to improve that. As a general comment, I find it very difficult to get the right balance between being aggressive and defensive. The deserting the leader is only one manifestation of that, although it's probably the worst offender.
I'll look into the long evaluation times also.
Thanks!
The replays are very helpful and I must say that I am not disappointed by what I am seeing. I mean, clearly Fred does things that are not ideal. But then, some of those I'm still working on, and overall I am not going for an AI that doesn't make mistakes and can beat a human player on even terms anyway. As long as he provides a different kind of challenge than the default AI and doesn't screw up really badly too often, I'm happy.
Yeah, Fred's still pretty bad at this "endgame" thing (both when losing and when winning, actually). Your replays are going to be great test cases for trying to improve that. As a general comment, I find it very difficult to get the right balance between being aggressive and defensive. The deserting the leader is only one manifestation of that, although it's probably the worst offender.
I'll look into the long evaluation times also.
Thanks!
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help requested with testing Fred - Freelands MP Custom A
Yeah, even after having played a few matches against Fred now, I still see him doing nothing wrong that is not already on your near term known issues list. And he is tremendously more intelligent and fun to play against than any other AI.
You may have achieved that right balance already; it is hard to make a case for Fred being consistently too aggressive or defensive in general. To illustrate, here is a replay of him against knalgans where he is actually a bit too aggressive, at least early on. (This specific case may derive from the known issue with resistances, since knalgans are a very resistance heavy faction in general.)mattsc wrote:I find it very difficult to get the right balance between being aggressive and defensive.
- Attachments
-
- fred_vs_knalgans.gz
- (26.42 KiB) Downloaded 846 times
Re: Help requested with testing Fred - Freelands MP Custom A
Thanks, that's great to hear!
Concerning the replay: I see what you mean. I don't think it's because of the resistances though, but due to one of those offensive/defensive balancing issues. In brief, when Fred advances, he considers whether the damage he'll receive at a location is acceptable. If so, move there, otherwise find a safer spot. In this case, the threshold for what's acceptable is too high. Problem is that when I set it lower, he becomes very scared and mostly just runs away. I think I need to make this threshold dependent on how far advanced the position is, how isolated the unis get etc. I'm not entirely sure yet how to do that yet, but even before your post this was the highest priority large/difficult item on my immediate task list.
In the meantime, I have fixed (most of) the issues Fred had with resistances, weapon specials like drain and the like. So he should do just fine against undead and knalgans now. Interestingly, he always seems to have the most problems against drakes, although I am not entirely sure yet why. I think it's a combination of the high mobility of drakes and the advancing problem described above.
With that, I have the following on my must-do list for the 0.14.1 release:
Concerning the replay: I see what you mean. I don't think it's because of the resistances though, but due to one of those offensive/defensive balancing issues. In brief, when Fred advances, he considers whether the damage he'll receive at a location is acceptable. If so, move there, otherwise find a safer spot. In this case, the threshold for what's acceptable is too high. Problem is that when I set it lower, he becomes very scared and mostly just runs away. I think I need to make this threshold dependent on how far advanced the position is, how isolated the unis get etc. I'm not entirely sure yet how to do that yet, but even before your post this was the highest priority large/difficult item on my immediate task list.
In the meantime, I have fixed (most of) the issues Fred had with resistances, weapon specials like drain and the like. So he should do just fine against undead and knalgans now. Interestingly, he always seems to have the most problems against drakes, although I am not entirely sure yet why. I think it's a combination of the high mobility of drakes and the advancing problem described above.
With that, I have the following on my must-do list for the 0.14.1 release:
- Fix that advancing problem described above
- Fix units abandoning the leader when enemy has many units closeby
- Do something about the long evaluation times toward the end (although I might not fix that entirely for 0.14.1)
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help requested with testing Fred - Freelands MP Custom A
Gave the drakes a shot (replay attached); I think I see what you mean, the extra mobility keeps his forces intimidated at a greater distance. Or maybe it's herpetophobia.mattsc wrote:Interestingly, he always seems to have the most problems against drakes, although I am not entirely sure yet why. I think it's a combination of the high mobility of drakes and the advancing problem described above.
The next release sounds juicy, definitely looking forward to it.
- Attachments
-
- fred_vs_drakes.gz
- (23.94 KiB) Downloaded 804 times
Re: Help with testing Fred - Freelands MP Custom AI (v0.14.1
Yeah, that must be it! And in some previous incarnation he really hated the fishmen. He dropped pretty much every other fight until he had gotten rid of those.Cold Steel wrote:Or maybe it's herpetophobia.
Thanks for the replay. One of the things that I would still like to get this AI to do better (and that is quite obvious in all the replays) is avoiding to expose individual units too much and then abandoning them, but I'm not sure if I'll entirely succeed with that. Still, I'm not unhappy at all with what I am seeing, and Fred's already much better at that than he used to be.
Thanks. I'm not sure that I would bet on it though. I think that all improvements from here on will be incremental, but hopefully you'll in the end notice small differences overall due to their combination. Thanks again for all the testing and all the replays, I really appreciate the help!Cold Steel wrote:The next release sounds juicy, definitely looking forward to it.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Fred (AI-demos) v0.14.1 is on the 1.12 add-ons server now with a few significant improvements over the previous version:
Overall, I believe that Fred does play better than in v0.14.0, although a lot of the improvements might not be easily noticeable.
Note that I will always show Fred's current status in the first post of this thread (which will get edited accordingly). In order to keep a record of past status and progress, this post (and any post on future updates) also contains the status of the previous version, with changes in the latest version shown via strikeouts, bold-face text etc.
- Units do not abandon their leader when faced with a large enemy force any more.
- Weapon specials, resistances, traits and abilities are now mostly taken into account correctly. (Of course, approximations need to be made otherwise the calculation time becomes too long, but those approximations should be pretty good by now.) As a result, the warning that Fred does not know how to play the undead has been removed.
- Many minor (and a couple major) improvements to attack and counter attack calculations that result in better attack behavior. Still some improvements to be made, but in some situations this is now much better than it was in 0.14.0.
- Some improvements to advancing and defensive holding. This is definitely not done yet.
- Significantly decreased evaluation time in situations when there are many AI and enemy units close to each other. As an example, the evaluation time for one of the situations in Cold Steel's first replay went from ~20 seconds to ~2 seconds on my computer. I don't know if this entirely fixes all situations, so please continue to let me know about really long evaluation times.
Overall, I believe that Fred does play better than in v0.14.0, although a lot of the improvements might not be easily noticeable.
Note that I will always show Fred's current status in the first post of this thread (which will get edited accordingly). In order to keep a record of past status and progress, this post (and any post on future updates) also contains the status of the previous version, with changes in the latest version shown via strikeouts, bold-face text etc.
Batch testing against the default AI
Previous and current known issues and future work (diff between v0.14.0 and v0.14.1)
v0.14.1 complete changelog
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Nice work!
I tried a few rounds against him with slightly more varied tactics. I feel that overall he did a good job of showing me the error of my ways when I used unusual tactics. In the "fred_vs_rebels" replay on turn 5, you can see him attack against a couple of units with a lone grunt during day, but I think this is because of a known issue you mentioned earlier.
One interesting surprise was his recruiting of naga during one particular match (replay: "fred_vs_courageous_fools") that I don't remember seeing him do before (or since). They were deployed fairly effectively as well.
I tried a few rounds against him with slightly more varied tactics. I feel that overall he did a good job of showing me the error of my ways when I used unusual tactics. In the "fred_vs_rebels" replay on turn 5, you can see him attack against a couple of units with a lone grunt during day, but I think this is because of a known issue you mentioned earlier.
One interesting surprise was his recruiting of naga during one particular match (replay: "fred_vs_courageous_fools") that I don't remember seeing him do before (or since). They were deployed fairly effectively as well.
- Attachments
-
- fred_vs_cowardly_brutes.gz
- (25.32 KiB) Downloaded 829 times
-
- fred_vs_courageous_fools.gz
- (22.42 KiB) Downloaded 877 times
-
- fred_vs_rebels.gz
- (16.44 KiB) Downloaded 876 times
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Oh, wow, that was quick, thanks! I hope that you're having some fun with that too.
Is it just me or is it because I was really hungry (and therefore a bit grumpy) when I watched the replays, or is Fred's performance not as convincing this time? I thought he should be doing a bit better than the previous version, but it almost looks the opposite. Either way, he clearly has a problem exposing individual units too much, not just during the turn you pointed out. As you said though, that part is not new and a combination of several of the known issues. I'll keep working on it...
Thanks!
Is it just me or is it because I was really hungry (and therefore a bit grumpy) when I watched the replays, or is Fred's performance not as convincing this time? I thought he should be doing a bit better than the previous version, but it almost looks the opposite. Either way, he clearly has a problem exposing individual units too much, not just during the turn you pointed out. As you said though, that part is not new and a combination of several of the known issues. I'll keep working on it...
The whole recruiting code was written by Alarantalara quite a while ago and has been in Fred (and Ron, in fact) pretty much unchanged since then. Since it seems to be doing pretty well overall (and yes, even throwing the occasional naga or other surprise at you), I am treating it as a black box for now and am simply using it as is.Cold Steel wrote:One interesting surprise was his recruiting of naga during one particular match (replay: "fred_vs_courageous_fools") that I don't remember seeing him do before (or since).
Thanks!
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Indeed, it is actually hard to go back to game types dependent on lesser AIs after you get used to one that plays basically like a person. And multiplayer matches seem to take around ten times longer to complete and cannot easily be saved part-way through for a later blob of free time. So even in a prototype state, I feel Fred currently offers the most interesting game play experience in BfW.mattsc wrote:I hope that you're having some fun with that too.
Plus I find AI in games a really interesting subject in general.
I didn't want to say after only a few matches, but this is possible. It felt like maybe he is a bit more aggressive in general than the last version, and this may lead to both more units getting isolated behind enemy lines and less massing of forces in strongholds or beyond attack range prior to an advantageous time of day. Additional matches will hopefully help tell the tale though.mattsc wrote: is Fred's performance not as convincing this time? I thought he should be doing a bit better than the previous version, but it almost looks the opposite.
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Great! I am glad to hear that you're enjoying this.
The tricky bit is, of course, figuring out how to tune things and your replays are going to be extremely useful with that. One of the nice things of the Lua AIs in Wesnoth is that you do not need to start from the beginning of the scenario/map in order to test new AI behavior. For example, I can take Turn 5 of the Rebels scenario and see how that grunt reacts to changes in the AI code (and figure out what the root cause of him storming forward like that is in the first place). So as I said, I'll keep working on it and hopefully a version or two down the road, there will be some actual noticeable overall improvement.
I have actually had this happen to me quite a few times. I've made what I thought were "great improvements", and when it comes down to real in-game testing, Fred ends up playing worse than before. Usually it turns out though that they really are improvements and that simply more tuning of the parameters is needed. I am confident that that is the case this time also - because I know what I changed and if used correctly, it simply cannot be worse than before (on average).Cold Steel wrote:I didn't want to say after only a few matches, but this is possible. It felt like maybe he is a bit more aggressive in general than the last version, and this may lead to both more units getting isolated behind enemy lines and less massing of forces in strongholds or beyond attack range prior to an advantageous time of day. Additional matches will hopefully help tell the tale though.
The tricky bit is, of course, figuring out how to tune things and your replays are going to be extremely useful with that. One of the nice things of the Lua AIs in Wesnoth is that you do not need to start from the beginning of the scenario/map in order to test new AI behavior. For example, I can take Turn 5 of the Rebels scenario and see how that grunt reacts to changes in the AI code (and figure out what the root cause of him storming forward like that is in the first place). So as I said, I'll keep working on it and hopefully a version or two down the road, there will be some actual noticeable overall improvement.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Another variable to consider is myself. I actually re-downloaded wesnoth when you announced the next phase of AI testing, following a hiatus of... however long it has been. So some of what you are seeing may not be fred getting worse but just me recalling how to play. As more testers come in this kind of thing should be less of an issue though, of course.mattsc wrote: Usually it turns out though that they really are improvements and that simply more tuning of the parameters is needed. I am confident that that is the case this time also - because I know what I changed and if used correctly, it simply cannot be worse than before (on average).
Having said that, here are a few more replays from me to further distort the situation.
- Attachments
-
- 14.1_vs_drakes.gz
- (17.61 KiB) Downloaded 821 times
-
- 14.1_vs_knalgans.gz
- (21.85 KiB) Downloaded 825 times
-
- 14.1_vs_undead.gz
- (20.92 KiB) Downloaded 808 times
Re: Help with testing Fred - Freelands MP Custom AI v0.14.1
Ah, I see, that might be true. Also, you are probably learning how to anticipate Fred's quirks to some extent and are adapting your play to that a little. Human players are very good at that as I have learned. You might want to start giving Fred a little bit more gold than yourself and see how that goes.Cold Steel wrote:Another variable to consider is myself. I actually re-downloaded wesnoth when you announced the next phase of AI testing, following a hiatus of... however long it has been. So some of what you are seeing may not be fred getting worse but just me recalling how to play.
Let's hope that the word will slowly spread and others will follow your example. These replays are really invaluable for me for figuring out what I need to be working on. I can't say thanks often enough!Cold Steel wrote:As more testers come in this kind of thing should be less of an issue though, of course.
Oh wow, that was bad what he did with the leader against the Drakes. But I think overall things are going pretty well for what Fred is at this point. I think it is becoming clearer and clearer that the biggest general mistake at the moment is sending individual units too far forward and then abandoning them there. I know some of the reasons for that, but that's going to be a hard one to get fully under control. In any case, I think that needs to be the main task for me to work on for 0.14.2.Cold Steel wrote:Having said that, here are a few more replays from me to further distort the situation.
In the meantime, I am running some more tests to figure out whether 0.14.0 was really better than 0.14.1 or if that's just our perception. I'll post about that in a day or two.
SP campaigns: Galuldur's First Journey (1.12 & 1.14) & Grnk the Mighty (1.10 & 1.12)
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on
AI experiments: Micro AIs (wiki, forum thread, known/fixed bugs), Fred, AI-demos add-on