Some love for the API

Posted in

#101 by micah686
2016-06-05 at 05:11
< report >Also, an API for anyone that can help me. Is there an easy way to get the last VNDB id that exists? I've tried using 'dbstats', which at the time of this writing shows 18764, but there are VNs with IDs of 19000 and over. Without the last vn id, it breaks my program.

My only other option would be to set up a scraper to scrape link, and find the highest revision, but I would prefer to not do that.

Yorhel explained to me how to do it.Last modified on 2016-06-05 at 06:32
#102 by tyr
2016-06-05 at 09:36
< report >link

Sorry for using a German Windows :p
It should be readable though.
#103 by micah686
2016-06-05 at 22:41
< report >Ok, try this release and post the log again:

The first one I gave you was just to narrow things down, since the code that contains the bug is over 1000 lines long. I SHOULD be able to replicate the error once you give me the newest log.
#104 by tyr
2016-06-05 at 22:46
< report >link
#105 by zolty
2016-06-16 at 18:05
< report >I got a few questions:

1. Does the API really not allow for a vn search filtered by tags?

2. Do commands such as get vn tags (id=[1,2,3,...]) result in less throttling than get vn basic,details,tags (id=[1,2,3,...])

It took me 4.5 hours to get all VNs released in 2013/2014 (around 2620 titles) using queries like these:

get vn basic,details,tags (released > "2010" and released <= "2012") {"results":25}
get release producers (vn ="751") (one for each vn so 25 of these)
get vn basic,details,tags (released > "2010" and released <= "2012") {"results":25, "page":2}
get release producers (vn ="4910") (one for each vn so 25 of these)

The most important thing for me are the tags so if it's faster to only get tags then I will probably change to that and I imagine each get release producers also adds to the throttling but is it significant?

The fact that there doesn't seem to be a way to get vns filtered by tags really hinders the abilities of a tag search application since one would need to get the tags of all vns first and then they would have to be kept updated through a grueling up to 32 hour process (extrapolating from the data above).
I can see how it could be resource-heavy looping through 18k titles but the website seems to do it pretty much instantly.
A solution could be to change the vns member in the tags dump to have the IDs of each vn with associated tag, rather than just the number of vns, a 24 hour update cycle seems perfectly fine to me and would obviously be better for the website if it generates that file rather than a bunch of people trying to query the website for 18k titles each.
#106 by yorhel
2016-06-16 at 18:18
< report >
I implement API features as people see a use for it. You're the first to ask about tag filters, so I'll see if I can implement it. Shouldn't be very hard to implement, nor very heavy on the server.

You're less likely to hit the SQL throttle if you ask for less information, but unless you use expensive searches or filters you're not very likely to hit that throttle at all. So in practice there's not much of a difference.
#107 by zolty
2016-06-16 at 19:10
< report >Alright thanks for the info.
I think both a filter on get vn command and a change to the tag dump would benefit, probably the latter more than the former since it would expose a lot of information without the need to query the servers and just the tag dump change would eliminate the need for a get vn filter since one could search the dump file for ids and then just query the vn info as usual.

I said earlier that the vns member could be changed but since some applications probably use it, it would be better to introduce a new member.

E: I just got all vns for 2011/2012 (2675~) and it only took 2 hours, are there peak times for the server?Last modified on 2016-06-16 at 21:37
#108 by yorhel
2016-06-19 at 09:46
< report >Tags filter added to the "get vn" command.

are there peak times for the server?
Not really, VNDB performance is pretty stable throughout the day as far as I can see. I did double the commands/minute throttle recently, if you did your earlier fetches before that update then that would certainly explain the difference.
#109 by zolty
2016-06-19 at 18:54
< report >That's great, thanks for the quick response, yorhel.
#110 by zolty
2016-06-27 at 00:37
< report >Interesting bug that baffled me for a minute, there was a VN release that had release date in Feb 29 2013, a day that never existed. r27995

Since the website allows 31 days, regardless of month/year on edits, programs using the API have to take that into account, I'm just going to modify any bad dates into the nearest real date.
#111 by yorhel
2016-07-02 at 09:30
< report >Nice find. I've updated the validation to disallow invalid dates on future release edits. Luckily there aren't many affected releases: link (these releases haven't been fixed yet).
#112 by zolty
2016-07-02 at 18:54
< report >I "fixed" the remaining release dates (couldn't find any concrete information on the internet though).
#113 by zolty
2016-07-06 at 20:18
< report >Hey yorhel, I'd like to ask, why do the tags in the tag dump contain the parent tags as opposed to the child tags?

It seems that child tags would be a better member because searching usually goes from parent to child, example:

You search for games with Female Protagonist (g134), you want to get all titles tagged with Female Protagonist or any of it's child tags like Strong Female Protagonist(g1869), etc.

The same information can be gotten by either parent or child tags but in order to do a search like the above, one would have to go over the tag dump several times to get all child tags:

taglist = tagdump.Where(tag.parents.Contains(134))
tier2list= tagdump.Where(tag.parents.Contains(any id in taglist ))
tier3list= tagdump.Where(tag.parents.Contains(any id in tier2list))
repeating until nothing more is found

If child tags were available, meaning not just direct children but all subtags, then it would be a single iteration
taglist = tagdump.Where(searchtag.children.Contains(tag.ID))

I understand that people weren't using the API for such things but I wonder how the website does it.Last modified on 2016-07-06 at 20:24
#114 by cloudedmind
2016-07-19 at 23:51
< report >Hello, I was wondering if anyone's created an external app or VNDB API that allows for multiple wishlists?
#115 by onkelsam
2016-07-26 at 18:05
< report >@micah686: Looks much better than my release (the code too). After my internship I realized my code was just a mess :).

@yorhel: Is there any way to get characters for multiple visual novels? Like "get character details (vn = " + VN-ID-Array + ")" ?
#116 by zolty
2016-07-27 at 10:33
< report >You can check by trying
"get character details (vn= [7,11,17])"
but it doesn't say you can in d11.

I stopped having array queries to make code easier and more unified, so that I only have a single function for getting vn info and just call it for each vn, I think it's much better than the previous list.skip(#).take(25) and string.join that I used to have, especially when getting info about a single vn may require multiple API commands, in my case, get vn, get release and sometimes get producer, and it will also be easier to modify in the future, if functionality to get staff is added.
#117 by onkelsam
2016-07-27 at 11:27
< report >Already tried that, but thanks nonetheless. Maybe that functionality could be implemented if it is not too much of a hassle.

Yes I agree, but it is (AFAIK) more efficient to query the characters for X number of VNs at once, as it is only one call to the database and not X calls.Last modified on 2016-07-27 at 11:28
#118 by yorhel
2016-07-27 at 14:58
< report >@115: i'll see what i can do (later), should be doable.

@116: using separate commands is certainly slower, and makes you more likely to hit the throttles.
#119 by yorhel
2016-07-31 at 02:29
< report >@115: "get character basic (vn=[..])" should work now.
#120 by zolty
2016-07-31 at 19:27
< report >Here's a screenshot of what I've been working on.
(some nsfw images are displayed because they were fetched before I started saving nsfw status)

I still have a lot that I wanna implement and lots of code cleanup to do but UI-wise, this is pretty close to what I want for the final product, along with some nicer backgrounds and colors.

I want to have as much space available for the VNs as possible so most information about what buttons do is in tooltips, clicking a VN brings up a small window with more details.
I will put it on github eventually.Last modified on 2016-07-31 at 19:30
#121 by micah686
2016-08-01 at 06:04
< report >So, I finished version 1.1 of my program. This should fix a lot of the crashes people were having with my previous one. It is a bit slower than the previous one at times, but I'm working on that.

And a thanks to onkelsam, while your code was not exactly clean, it did provide a base for me to strip down, and work off of.

also, @zolty, would you be willing to share your code with me, possibly privately? It looks nice, and I would like to see about using some of the features that you have into my project.
#122 by tyr
2016-08-01 at 08:13
< report >@121. Still crashes during login. This time I can see roughly 40 novels in the left list before it happens.

Debug says:
Exception Found:	Type: System.ArgumentOutOfRangeException
Class File: UserVnListViewModel.cs
Method Name: GetVnData
Parameter name: index
Source: mscorlib
StackTrace: at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
at System.Collections.Generic.List`1.get_Item(Int32 index)
at Visual_Novel_Manager.ViewModel.UserVnListViewModel.<GetVnData>d__50.MoveNext()
#123 by zolty
2016-08-01 at 11:58
< report >@micah686
You'll be able to see my code when I put it on github maybe in one or two weeks.
#124 by onkelsam
2016-08-01 at 18:41
< report >@yorhel: Thank you, works like a charm.

Now, after working on and off on a newer version for the last few months, I can finally release the VNDBUpdater.

It supports syncing with, an automatic file indexer (I hope it works for you :)) and relatively clean code.


Everyone is of course free to take parts of my code and use it. I created this version mainly for myself, so the features included are mostly what I wanted (and the GUI still doesn't look good...).

You probably have to start it with admin privileges. If a prompt pops up to allow network access, remove all check marks and click OK (that is caused by using 'Redis' as the database).Last modified on 2016-08-01 at 19:52
#125 by micah686
2016-08-02 at 02:01
< report >Ok Tyr, I THINK I fixed it., so it won't crash. The program was overwriting the array holding the list of vns, so it probably crashed when trying to access one of the overwritten VNs.

Anyway, try this version out, and tell me if you still have the issue. Also, tell me if it DOES work, so I can update my github page with the fix.