Sign In To Proceed i2v15

Don't have an ? 1a2w41

osu! to create your own !
forum

Yet another beatmap mirror + data API. 1q2rh

posted
Total Posts
73
Topic Starter
Hello, I have decided to open up my mirror to public. Here you will find details how to use it and information about it.

General info:
  1. Mirror is updated every 6 hours
  2. Mirror contains only ranked beatmaps
  3. Server can at maximum 300mbps
How to access mirror and beatmaps:
  1. Direct link to files on the web: https://osu.hexide.com/list/
  2. Desktop application: https://mega.nz/#!pnpyTJjQ!yVtUeoGMbwz4HjJOqqb3lrfw1Me9zHQx68ZRlODC8y0 (-Source code-)
API LOCATION:
https://osu.hexide.com/

Statistical data:
SPOILER
Global network utilisation (maximum 300/300mbps):
Soon
Added to the osu! Projects Compilation thread. Keep the good work!!!
Topic Starter
Started updating api to extend functionality. http://api.osu.miz.hexide.com/
Meantime search page will not be working.
Topic Starter
Started working on small c# for multiple simultaneous beatmap s. It's still in extremely early stages https://github.com/Mizore/osu-mirror
Implemented few metadata lookups into api http://api.osu.miz.hexide.com/#/get/metadata/id ; http://api.osu.miz.hexide.com/#/get/metadata/id/raw

Search calls are coming soon.
Primula_old
This is really nice, thank you.
Will implement it into osu!np soon~
Topic Starter

Primula wrote: 163p2n

This is really nice, thank you.
Will implement it into osu!np soon~
That would be great! If you will have any requests/recommendations, feel free to poke me.
Topic Starter
Coded search api. There might still be some future updated, but that wont be soon. Currently it works pretty well.
http://api.osu.miz.hexide.com/#/get/search documentations are still being written.

Also, please don't kill server. :D
Very cool ftp, fast speed. I'm going to everything xD
Topic Starter

power007 wrote: 1l2e1g

Very cool ftp, fast speed. I'm going to everything xD
Cheers. :)
Topic Starter
Also did some big updates to how search works now.
In a nutshell, you need to one or more search parameters. Search query will look like "/search/[param]/[param]/[param]".

Each parameter is made of 4 peaces: "[table].[column].[operand].[value]" some examples of this would be:
  1. metadata.name.like.fade
  2. maps.size.gt.100000000
  3. maps.title.like.dance
Possible operands are:
  1. eq - equal
  2. lt - less than
  3. lteq - less than or equal
  4. gt - greater than
  5. gteq - greater than or equal
  6. like - like
There are also some limitations like:
  1. You cannot use lt, lteq, gt, gteq with VARCHAR columns
  2. You cannot use like with INT and DOUBLE columns
  3. You can only use eq with ENUM columns
For list of all columns, please refer to: http://api.osu.miz.hexide.com/#/db/maps and http://api.osu.miz.hexide.com/#/db/metadata

Hopefully this will be useful for someone. :)
Topic Starter
If any graphic artists would be willing to help me out, it would be amazing. I need someone to change some colors in few bitmaps.
Topic Starter
Topic Starter
Allot of progress on c# app for beatmap s.
Here are some images:
http://miz.hexide.com/XSgE.png
http://miz.hexide.com/bUdy.png

More coming soon! :)

If someone is interested in testing it out:

Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

For paranoid: https://github.com/Mizore/osu-mirror
I like the for paranoid part, going to start doing some icons for you today. :)
Topic Starter
Topic Starter
past few day change digest:

Fixed registry issue where data would be be saved to incorrect location.
Added for enter clicks on text boxes to perform search instead of clicking 'Search' button.
Files no longer will contain first capital letter in their extension.
Files will no longer be taking up more space than they should Issue #1
Added sorting for standard list-view.
Added MP3 file .
audio files will no longer have numbers at the start of their names.
Added for changing count of threads used for ing file ( currently only changeable in registry )
Updated files:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

Also, GitHub page if you want to track progress: https://github.com/Mizore/osu-mirror
Topic Starter
Bringing mirror down for some core improvements, could take few hours.
Back

Some more updates: Should be done within few hours.
Topic Starter
Looks like everything is updated and restored :D , there are some major changes:
  1. Now ALL beatmap related images and other files will be under /content rather than /preview
  2. API is no longer friendly, what i mean by that, calling / on some beatmap will not start with original beatmap name
  3. Removed /preview/mp3/full/
  4. Removed /preview/image/tiny,small,medium,large (only full and custom options are left)
And as always, most recent file:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

https://github.com/Mizore/osu-mirror
the exe doesnt work just crashes...

Takeda Shingen wrote: 3j1f30

the exe doesnt work just crashes...
and I thought I'm the only one...
this happens after clicking search
Topic Starter
Hmm... Made small change in code as potential fix, how ever not sure if it will help.

Updated files:
Main EXE: http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
Debug symbols: http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

P.S. There will be no search results if you will not enter at least one letter into any search field.
working now.
(and found duplicate entry- http://puu.sh/3v6qW.png)
Topic Starter
Thanks for reporting, I could find actual bug in code. But what I found was that search was performed during beatmap update, since duplicate records are not in database. It should not contain duplicate result for any other similar searches, except for that exact one, since its cached.
It will clear up eventually :)

1. Yup, found few other duplicate beatmaps too. Will be fixed in few moments.
2. Fixed, those duplicate entries from search will go away once cache entry expires ( could take a while :P )
My friend can't run this program under 32Bit system(Win XP)
Topic Starter
Its .NET 4.5 app. It does not seem that it was released for WinXP.
So the only suggestion I could give now, is to upgrade.
Topic Starter
Added option to view beatmap details on this site:


If anyone has suggestions, ideas; feel free to share. If possible I will implement them.
Topic Starter
Added menu bar.
Moved settings to separate window.
Added ability to change concurrent limit.
Added a bunch of links to 'Help' menu.
Recommended concurrent s are recommended to be kept bellow 6. Exception would be for people who have extremely good connection to beatmap mirror ( 8MB/s and up ).
Mine is set at 30 because I have direct 1Gbit connection to server.

And as always, updated files at:
http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb

http://miz.hexide.com/dDld.png
http://miz.hexide.com/AbD9.png
Topic Starter
Added option to disable question window when closing app.
Added finished animations to queue.
Some minor tweaks that I forgot.
Debug build: ( provides useful information in case of crashes )
http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb
Release: ( does not display debug console, slightly higher performance )
http://miz.hexide.com/mirror/Beatmap%20 ... elease.exe
Topic Starter
Spent good while configuring nginx and added some more feature to main api: ;)
  1. Raw access to beatmap file contents: /beatmaps/12191/content/raw /beatmaps/12191/content/raw/background.jpg
  2. File hash calculations for raw beatmap files: /beatmaps/12191/content/raw/hash/noise/1.png
  3. Raw file layout: /beatmaps/12191/content/raw/layout
  4. Now all invalid requests should receive API friendly 400 response.
  5. All responses are now cached for 24 hours.
Notes:
  1. If requested beatmap was not extracted prior to accessing raw files, they will be extracted once and "Location" header will be served to request.
  2. Documentations coming soon.
Topic Starter
It seems that there are allot of s that don't have agent set. Who is writing apps/scripts/services that use this api, please specify agent ( something relevant ) and if its public projects that has thread on this forum, please specify thread in Referrer header like "/t/137156".

Preferred minimal http request structure ( this is example ):
GET /search/maps.title.like.night HTTP/1.1
Host: api.osu.miz.hexide.com
-Agent: Osu!Mirror
Referer: /t/137156
Connection: Keep-Alive
Topic Starter
Added some basic stats: http://api.osu.miz.hexide.com/stats
ALLOT MORE coming soon.
Topic Starter
And I broke everything. Api responses no longer contain "status" => "....".
Response code 200 = everything is good. 300 >= error.

Documentations will be updated in a moment.
Topic Starter
Updated mirror app to new api structure.

Debug build: ( provides useful information in case of crashes )
http://miz.hexide.com/mirror/Beatmap%20Mirror.exe
http://miz.hexide.com/mirror/Beatmap%20Mirror.pdb
Release: ( does not display debug console, slightly higher performance )
http://miz.hexide.com/mirror/Beatmap%20 ... elease.exe
this is an excellent, well documented service. thank you for hosting it!
Topic Starter
Well looks like project is pretty stable state.
All features that I had in mind are implemented and documented.

I also wrote small php class a while ago that could be used with api:
Class: https://gist.github.com/Mizore/5939745
Usage: https://gist.github.com/Mizore/5939747
A bug report (I'm not sure if you have another place for these:
When doing a GET /beatmaps (http://api.osu.miz.hexide.com/beatmaps), the documentation says the result should include:
		name: "31066 DJ YOSHITAKA - SEED",
title: "DJ YOSHITAKA - SEED",
type: "osz",
However, the API actually responds with, for all the maps:
        "name": "1 Kenji Ninuma - DISCO PRINCE",
"title": null,
"type": "Kenji Ninuma - DISCO PRINCE",

For some reason "title" is always null and "type" contains the beatmap title (and there is no way to get the type information).
It doesn't seem like this happens on other API requests, but I haven't checked them all.

Thanks a lot for this service; it's very well documented!
Topic Starter
Whoops. Fixed. Issue was that i forgot to place comma in between 'title' and 'type' columns :v

It might still show broken version, should take up to hour for cache entry to expire.
Topic Starter
Also fixed minor big that caused all images returned from api to be full size.
Fixed issue where beatmaps with no wallpaper would still return status 200 response.
Now instead of 404 response on beatmap images, stock icon would be returned (also resized and cropped)
Topic Starter
If anyone is curious in API source code: http://git.hexide.com/Miz/osuapi

Also threw in database dump ( no data ).
http://git.hexide.com/Miz/osuapi/commit ... 90cd51dbb6
Topic Starter
Just realised that beatmaps were not updated for past 10 days (last successful update was on 2013-11-12 00:00:20).
Fixed issue and following beatmaps were updated:
Avail_old_1
It's kind of broken :(
MP3s opens the queue window, puts the song in it, and closes it. (Both on debug and release.)
Beatmap crashes the app in debug, freezes the app in release. :(
Topic Starter

Avail wrote: 1r5o4y

It's kind of broken :(
MP3s opens the queue window, puts the song in it, and closes it. (Both on debug and release.)
Beatmap crashes the app in debug, freezes the app in release. :(
Possibly due to missing directories where s pointed to. That is the only thing that comes to my mind at the moment.

You could try using latest version: http://miz.hexide.com/mirror/BeatmapMir ... 3adfd2.exe
Avail_old_1
Everything is pointed to my s so I doubt it's a missing dir.
EDIT: Same thing happens with latest build..
EDIT2: Just noticed ( a couple hours late ) how the app actually ed the .mp3, well, a blank one.
Topic Starter

Avail wrote: 1r5o4y

Everything is pointed to my s so I doubt it's a missing dir.
EDIT: Same thing happens with latest build..
EDIT2: Just noticed ( a couple hours late ) how the app actually ed the .mp3, well, a blank one.
Does http://api.osu.miz.hexide.com/ open up properly for you?
Avail_old_1

Hexide wrote: 6l445p

Does http://api.osu.miz.hexide.com/ open up properly for you?
Yup. Works fine.

? :(
It crashes every time I try to open it :(
Topic Starter
After installing Framework 4.5 it works.
Thanks.
Topic Starter
Small update:
Removed details window (because it was not about to get finished any time soon).
Added double click .
Added option that would prevent/allow queue window to be automatically hidden when is finished.
Added bit more debug logging.
Updated address to project repository.
Release build: http://miz.hexide.com/mirror/BeatmapMirror.8077c5ce9.exe
Git: http://git.hexide.com/Miz/osumirror/compare/d903adfd2...8077c5ce9
Topic Starter
Whoops, looks like last successful beatmap update was at 2013-12-26 00:00:11. I updated php and forgot to install one module, which led to beatmaps not updating properly.

All is fixed. Might take couple hours to grab all new beatmaps.
I have a few things that could be improved:

5513 doesn't seem to be valid. When I it I get an md5 of 57a75e4030beb6174089c6f91fcf9a5d while the info says 3fcfe7eae40a8fa86e2d151ee536735 (also md5 of the valid map from http://osu-ppy-sh.cinevost.com/s/5513)
Edit: 19491, 8442, 14431, 17796, 41482, 17796, 18879, 12292, 13422, 1735 and 2533 all look invalid. Am I doing something wrong?

There are a number of spelling mistakes. If you want, I can make a list.

On http://api.osu.miz.hexide.com/doc#/get/beatmaps/id it says
"Request data: [0-9]+ Can be any integer, ranked beatmap id"
which is correct, but on every other page (e.g. http://puu.sh/6zh7e.png) it just says "beatmap id" which led me to use id instead of ranked_id everywhere.
Topic Starter
Right, they don't seem to be entirely correct.
Ill fix it asap ( 3 - 4 days hopefully, since I'm in the middle of moving out )
Hey, first of all I just wanted to say great job on the API. It's extremely helpful for me, having data request via MD5 hash.

One problem I ran into though, is when using the metadata request (Example: http://api.osu.miz.hexide.com/metadata/9e63d9a0b139c5953101e5eaf77af809), 'm_beatmap' and 'm_beatmapset' don't seem to be displaying (Image example).

Any chances this could be fixed? Would be extremely helpful so I only have to make a single request rather than multiple per map.

Thanks!
Topic Starter

Woddles wrote: 3k6t3o

Hey, first of all I just wanted to say great job on the API. It's extremely helpful for me, having data request via MD5 hash.

One problem I ran into though, is when using the metadata request (Example: http://api.osu.miz.hexide.com/metadata/9e63d9a0b139c5953101e5eaf77af809), 'm_beatmap' and 'm_beatmapset' don't seem to be displaying (Image example).

Any chances this could be fixed? Would be extremely helpful so I only have to make a single request rather than multiple per map.

Thanks!
Hey, that data is missing because it was not present in beatmap at the time of parsing. It will show up when will become available. Also multiple requests is not a problem.
Topic Starter
Sorry for down time, ISP blew up.
Topic Starter
Fixed issue where all output which was returned from cache did not have 'Access-Control-Allow-Origin'.
Now all responses have it ( its set to `*` ).
Topic Starter
Just realized one feature was never documented or announced.
API s AND OR logic for filtering, for example of ed queries:
  1. where m_title like dark OR m_artist like m2u - Wide range of beatmaps, containing beatmaps containing only `dark` in title or `m2u` as artist
  2. where m_title like dark AND m_artist like m2u - Only one result, beatmap which contains `dark` in title and `m2u` as artist
Also added into docs: http://api.osu.miz.hexide.com/doc#/get/search
Topic Starter
Fixed issue with new php5 version which caused beatmaps not to be updated.
These (http://miz.hexide.com/IKgZ8MaU.html) beatmaps could have been out of date for up to 7 days.
I can't click on tools :c
Topic Starter

marcostudios wrote: 2j1961

I can't click on tools :c
Nothings there. Never actually anyone requested anything.
Topic Starter
API LOCATION CHANGE:
https://osu.hexide.com/
Hey Hexide, I can't access to http://osu.miz.hexide.com/beatmaps/ anymore since a month or so.
Is it a oversight or did you do it on purpose?
Topic Starter

Hexide wrote: 6l445p

It's at https://osu.hexide.com
Nope sorry I didn't specify, I meant the beatmaps mirror.
There are just the API there (or am I missing something?).
Topic Starter

Kirito_Beater wrote: 43596x

Hexide wrote: 6l445p

It's at https://osu.hexide.com
Nope sorry I didn't specify, I meant the beatmaps mirror.
There are just the API there (or am I missing something?).
Oh, the list for direct s. Will enable it in few moments.
Topic Starter

Hexide wrote: 6l445p

At https://osu.hexide.com/list/
Wow that was fast!
Thank you,
Have a nice day. :)
Hey, have you updated your mirror?
It seems that it's missing newer maps like this.
Topic Starter
Hi, yes, it is not synchronizing with main beatmap repo. Will fix later today.
Topic Starter
Back on online.

https://osu.hexide.com/doc
https://osu.hexide.com/stats

Beatmap archive is massively out of date, will look into synchronizing it later today.
Topic Starter
Recovered.
Beatmaps still syncing, so far 84k done... ~25% done.

- Cleared stats
- Added missing documentation for /cache/
- Fixed issue where no JSON would be returned for /beatmaps/[0-9]+/content/raw/layout

Also nice to see that opsu! client is still using this API :)
Topic Starter
Pushed small update to desktop application:
https://git.hexide.com/miz/osumirror/co ... ..6f5d9a09

If someone wants a binary build:
https://mega.nz/#!pnpyTJjQ!yVtUeoGMbwz4 ... 8ZRlODC8y0
Topic Starter
Updated API, added to filter beatmaps by:
* maps.hash_md5
* maps.hash_sha1
* metadata.hash_md5
* metadata.hash_sha1

* metadata.a_beatmapset_id
* metadata.a_beatmap_id
* metadata.a_approved
* metadata.a_total_length
* metadata.a_hit_length
* metadata.a_bpm
* metadata.a_max_combo

These all values come from original API (https://github.com/ppy/osu-api/wiki)

Something that app developers might want to implement is filter by: a_approved
Here are few examples:
* https://osu.hexide.com/search/maps.rank ... oved.eq.-2 - graveyeard
* https://osu.hexide.com/search/maps.rank ... roved.eq.1 - ranked

P.s. Data update is still in progress for beatmaps, not all maps have metadata.a_* fields populated.
Please sign in to reply.

New reply 18684e