Doctor Posted March 15 Report Share Posted March 15 (edited) So, I'm semi-abandoning a project I started but am probably not going to finish anytime soon. But I dumped enough hours into it, and feel it has enough merit, that I am going to share the source code in case someone else thinks this is cool and wants to work on it. What is it? A replacement for enbmaps.de and all the other mapping sites we have. I wanted it to show where mobs spawned, and where roid fields were, along with the mining results from every zone (see the wiki on pages like Glenn, to see what I mean by 'mining results', the list of ores found in each sector). Right now, I have all the sectors and all the navs (that I could find) imported. Look in the 'initialization data' folder. No mobs/roids/gravity wells have been added yet. The annoyance of logging ore fields in a way that didn't drive me nuts (and the tedium of logging mobs) made me not want to bother...for about 2 weeks now. "What's the license" You may not use this software for commercial gain. No ads, no donations, no other revenue streams. If you break even with hosting and publicly (no request, publicly, on a normal page anyone can view) display that your operating it as non-profit, that's fine. It must remain open source in its entirety. Every image, every line of code, every byte of data in the database must all be accessible without authorization or signup. The code cannot be placed on github or any other site which intends to ingest the code into AI tools or otherwise monetize the code hosted there. Running your own git instance and hosting from there is fine. Any future license must require and honor these terms. Meaning, you cannot repackage this and change the license to MIT or something else. These terms of license must be in all downstream forks. For running a production site For an actual production site, you'll probably want to use mysql instead of sqlite, but sqlite is great for development and the code changeover from one to the other isn't bad. Mostly it's replacing ? with %s in a bunch of places. Well...the binding process is completely different, but meh. I didn't write it using mysql, again, if zen4's Promontory 21 hadn't been the shitshow it was, I'd have written it to use mysql. Also, you'll probably want to remove the Command Bar entirely from a production site. This tool wasn't really built to work like a wiki. It was meant for 1-2 people who are highly trusted to update the DB, then for it to mostly be a read-only experience. The idea behind adding mobs: Fly around with two chars. A JD and a PW are optimal. JD has 3 beams, 1 Energy, 1 EMP, 1 Plasma. PW has 3 guns, one loaded Explosive, 1 loaded Impact, 1 loaded Chemical. Telescopium on the JD DO NOT USE ANY RESIST DEBUFFS OF ANY KIND! Not even gravity link. Click on the 'Add Mob' button, shoot with energy laser, say it does 145 (-50) damage. You'd type in 145 for the Damage Component and -50 for the Resist Component. There's a blub on the page that explains all this. Fire 1 round from all the other weapon types and enter the damage/resist components. For positive numbers just enter the number without a + sign, so +500 is just 500. The page is smart enough to know that 145 (-50) meant you did 195 dmg, and 1000 (+500) meant you did 500. The page will do the math on its own to determine the mob resists and store them in the DB. A checkbox needs to be added for whether or not the mob is default hostile. "All this is in the wiki!" Yeah, but seeing it on a map is easier, and for things like mobs and ore, it helps to visually see where things are. Sure, there's Titanium in KV, but *where* in KV. Oh, only 2-3 of the 17 fields there. Cool. Now I know where to warp. "This javascript is written by a moron" Yeah. this is my first major javascript attempt. I was making this a web 2.0 page (so, one page load, then only DOM changes after) purely for practice/learning. You'll note I don't import jquery or any libraries. This is all hand-written baisc javascript. Maybe I'm a masochist. Maybe I just like learning. *shrug* "How did you get the navs?" Log into the game (ON ONE CHARACTER, DO NOT MULTIBOX FOR THIS unless using separate installs), enter a sector, Create a private channel (mostly to avoid spamming others) Target a 'normal' nav, press ctrl+t (I remapped it to b for easier pressing) Press n to go to the next visible nav Press ctrl+t Repeat until all visible navs are reported in chat. Don't worry about duplicates, the code below is smart enough to remove them. type 'hidden' in your private channel press 'x' to target the nearest hidden nav (move away if asteroids are in view) Press ctrl+t Press n to go to the next hidden nav Repeat until all hidden navs are dumped Close game Run script below on your chat.log file Copy-paste output into the "Add Navs" button, after you check the 'JSON Import' checkbox I imagine a script similar to this will be useful for import mobs/roids from chat as well. I did not do the work to make this a thing though. Also, I did not learn sectors could be asymmetric (like Grissom, where it's box is -450|575|-525|650) until I was almost done adding sectors, so roughly 40 of the 69 sectors in the game should be double-checked for asymmetry. sector_ids 1-53 are suspect, basically. Xipe Totec was the first asymmetric zone I remember finding. After Xipe, I made sure to check all 4 dimensions instead of assuming symmetry. --Parse_navs.py-- #!/usr/bin/python3 import argparse import json import re """THis list is also defined in functions.php and map_script.js, update all 3! $NAV_TYPE_MAP_BASE = [ "Normal Nav" => 0, "Hidden Nav" => 1, "Normal Gate" => 2, "Faction Gate" => 3, "Normal Station" => 4, "Faction Station" => 5, "Planet" => 6, "Landable Planet" => 7, "Weft" => 8, "Extended Weft" => 9, "Ore Field" => 10 ]; """ def parse_log(file_path, channel_name, player_name): results = {} sector_pattern = re.compile(r"We have entered (.+?) Sector \((.+?)\)") target_pattern = re.compile(r"\[(\d+)\] " + re.escape(player_name) + r": Target '(.+?)' at \(([-\d.]+), ([-\d.]+), ([-\d.]+)\)") hidden_nav_pattern = re.compile(r"\[(\d+)\] " + re.escape(player_name) + r": hidden") current_sector = "" hidden_nav_mode = False with open(file_path, 'r', encoding='latin-1') as file: for line in file: sector_match = sector_pattern.search(line) if sector_match: sector_name = sector_match.group(1) system_name = sector_match.group(2) current_sector = sector_name #print(f"We have entered {sector_name} ({system_name})") hidden_nav_mode = False if hidden_nav_pattern.search(line): hidden_nav_mode = True target_match = target_pattern.search(line) if target_match: target_channel = target_match.group(1) target_name = target_match.group(2) nav_type = 0; if target_name.startswith("Sector Gate ") or target_name.startswith("Accelerator") or target_name.startswith("System Gate"): nav_type = 2; if hidden_nav_mode: nav_type = 1 if target_channel == channel_name: x_coord = int(round(float(target_match.group(3)),0)) y_coord = int(round(float(target_match.group(4)),0)) z_coord = int(round(float(target_match.group(5)),0)) if current_sector not in results: results[current_sector] = {} #Assume normal navs until we have some way to mark other things automatically. coord_pack = [x_coord, y_coord, z_coord, nav_type] if target_name not in results[current_sector]: results[current_sector][target_name] = [coord_pack] elif coord_pack not in results[current_sector][target_name]: results[current_sector][target_name].append(coord_pack) #print(f"Target '{target_name}' at ({x_coord}, {y_coord}, {z_coord}, {nav_type})") #print(json.dumps(results, indent=4)) #print(results) print(format_output(results)) def format_output(data): formatted_output = "{" for key in data: formatted_output += f'"{key}": {{\n' for nav in data[key]: formatted_output += f'\t"{nav}": {data[key][nav]},\n' formatted_output = formatted_output[:-2] + "},\n" # Remove the trailing comma and newline character for the last line formatted_output = formatted_output[:-2] + "}\n" return formatted_output if __name__ == "__main__": parser = argparse.ArgumentParser(description="Parse log file") parser.add_argument("file", help="Path to the log file") parser.add_argument("--channel", default="100", help="Channel name (default: [100])") parser.add_argument("--player", default="Doctorje", help="Player name (default: Doctorje)") args = parser.parse_args() parse_log(args.file, args.channel, args.player) --parse_ores.py-- #!/usr/bin/python3 import argparse import json import re import sqlite3 import sys RESOURCE_SOURCE_MAP = { "Copper Ore": ["Rock", 1], "Crude Graphite": ["Hydrocarbon", 1], "Crude Nickel": ["Rock", 1], "Diridium Crystal": ["Crystal", 1], "Hydrogen": ["Gas", 1], "Iron Ore": ["Rock", 1], "Lead Ore": ["Rock", 1], "Lithium Ore": ["Glowing", 1], "Magnetite Ore": ["Rock", 1], "Mordanite": ["Rock", 1], "Nitrogen": ["Gas", 1], "Oxygen": ["Gas", 1], "Potash": ["Glowing", 1], "Raw Agate": ["Crystal", 1], "Raw Beryl": ["Crystal", 1], "Raw Black Tazeron": ["Hydrocarbon", 1], "Raw Bloodstone": ["Crystal", 1], "Raw Blue Tazeron": ["Hydrocarbon", 1], "Raw Crude Coal": ["Hydrocarbon", 1], "Raw Green Tazeron": ["Hydrocarbon", 1], "Raw Jasper": ["Crystal", 1], "Raw Onyx": ["Crystal", 1], "Raw Red Tazeron": ["Hydrocarbon", 1], "Raw Topaz": ["Crystal", 1], "Raw White Tazeron": ["Hydrocarbon", 1], "Raw Yellow Tazeron": ["Hydrocarbon", 1], "Sand": ["Hydrocarbon", 1], "Sponge Chloride": ["Crystal", 1], "Sulfates": ["Glowing", 1], "Tar": ["Hydrocarbon", 1], "Uranium Ore": ["Glowing", 1], "Zinc Ore": ["Rock", 1], "Aluminium Ore": ["Rock", 2], "Brominite": ["Glowing", 2], "Cadmium Ore": ["Glowing", 2], "Carbon Dioxide": ["Gas", 2], "Crude Oil": ["Hydrocarbon", 2], "Germanium Ore": ["Rock", 2], "Helium": ["Gas", 2], "Hermesite": ["Rock", 2], "Magnesium Ore": ["Rock", 2], "Methane": ["Gas", 2], "Molybdenum Ore": ["Rock", 2], "Phosphates": ["Glowing", 2], "Plutonium Ore": ["Glowing", 2], "Quartz Crystals": ["Hydrocarbon", 2], "Raw Amethyst": ["Crystal", 2], "Raw Coal": ["Hydrocarbon", 2], "Raw Garnet": ["Crystal", 2], "Raw Moonstone": ["Crystal", 2], "Raw Sunstone": ["Crystal", 2], "Raw Tourmaline": ["Crystal", 2], "Raw Turquoise": ["Crystal", 2], "Tin Ore": ["Rock", 2], "Zircon": ["Rock", 2], "Alanite": ["Glowing", 3], "Barite": ["Glowing", 3], "Boronite": ["Glowing", 3], "Caesium Ore": ["Rock", 3], "Calcite": ["Glowing", 3], "Cobalite": ["Rock", 3], "Fluorine": ["Gas", 3], "Gallium Ore": ["Rock", 3], "Indium Ore": ["Rock", 3], "Light Crude Oil": ["Hydrocarbon", 3], "Obsidian": ["Hydrocarbon", 3], "Polonium Ore": ["Glowing", 3], "Raw Alexandrite": ["Crystal", 3], "Raw Anthracite": ["Hydrocarbon", 3], "Raw Citrine": ["Crystal", 3], "Raw Lapis Lazuli": ["Crystal", 3], "Raw Malachite": ["Crystal", 3], "Star Iron Ore": ["Rock", 3], "Tungsten Ore": ["Rock", 3], "Anubium Ore": ["Rock", 4], "Argon": ["Gas", 4], "Californium Ore": ["Glowing", 4], "Ceresite": ["Glowing", 4], "Galactic Ore": ["Rock", 4], "Herculinium Ore": ["Rock", 4], "Manganese Ore": ["Rock", 4], "Meteoric Sand": ["Hydrocarbon", 4], "Neon": ["Gas", 4], "Raw Black Opal": ["Crystal", 4], "Raw Centauricite": ["Rock", 4], "Raw Fire Opal": ["Crystal", 4], "Raw Flawless Garnet": ["Crystal", 4], "Raw Ruby": ["Crystal", 4], "Saganite": ["Glowing", 4], "Silver Ore": ["Rock", 4], "Solar Sweet Oil": ["Hydrocarbon", 4], "Vanadium Ore": ["Rock", 4], "Andromesite": ["Hydrocarbon", 5], "Bastinium Ore": ["Glowing", 5], "Brood Oil": ["Hydrocarbon", 5], "Chromium Ore": ["Rock", 5], "Gold Ore": ["Rock", 5], "Hades Blood": ["Hydrocarbon", 5], "Halon": ["Gas", 5], "Hawkinsite": ["Glowing", 5], "Minervite": ["Glowing", 5], "Radium Ore": ["Glowing", 5], "Radon": ["Gas", 5], "Raw Black Pearl": ["Crystal", 5], "Raw Capellicite": ["Crystal", 5], "Raw Heartstone": ["Crystal", 5], "Raw Sapphire": ["Crystal", 5], "Raw Skystone": ["Crystal", 5], "Rhodite": ["Rock", 5], "Titanium Ore": ["Rock", 5], "Zalmoxium Ore": ["Rock", 5], "Adamantine Ore": ["Rock", 6], "Charon's Dust": ["Hydrocarbon", 6], "Curium Ore": ["Glowing", 6], "Discordite": ["Glowing", 6], "Hafnium Ore": ["Rock", 6], "Homerite": ["Rock", 6], "Krypton": ["Gas", 6], "Mirandium Ore": ["Rock", 6], "Osirium Ore": ["Rock", 6], "Oxium Ore": ["Rock", 6], "Platinum Ore": ["Rock", 6], "Raw Anthenicite": ["Hydrocarbon", 6], "Raw Charon Crystal": ["Glowing", 6], "Raw Emerald": ["Crystal", 6], "Raw Flawless Ruby": ["Crystal", 6], "Raw Icy Pearl": ["Crystal", 6], "Stygian Blackwater": ["Hydrocarbon", 6], "Xenon": ["Gas", 6], "Apollonite": ["Glowing", 7], "Brucite Ore": ["Glowing", 7], "Celestial Ore": ["Rock", 7], "Chalcophanite": ["Glowing", 7], "Conorite": ["Rock", 7], "Cupidite": ["Glowing", 7], "Demeter's Tears": ["Hydrocarbon", 7], "Horusium Ore": ["Glowing", 7], "Inderite": ["Rock", 7], "Iridium Ore": ["Rock", 7], "Leonite": ["Hydrocarbon", 7], "Neutronium Ore": ["Glowing", 7], "Niobite": ["Rock", 7], "Raw Barite": ["Glowing", 7], "Raw Diamond": ["Crystal", 7], "Raw Eye Stone": ["Crystal", 7], "Raw Firerock": ["Crystal", 7], "Raw Galactic Rimstone": ["Crystal", 7], "Raw Hadecite": ["Hydrocarbon", 7], "Stojsavline": ["Gas", 7], "Stygian Blacksand": ["Hydrocarbon", 7], "Tantalum Ore": ["Rock", 7], "Vaneon": ["Gas", 7], "Wexeon": ["Gas", 7], "Abyssian Dust": ["Hydrocarbon", 8], "Ambrosia Crude": ["Hydrocarbon", 8], "Boragon": ["Gas", 8], "Duplium Ore": ["Rock", 8], "Emperion": ["Gas", 8], "Helvatha": ["Hydrocarbon", 8], "Idunium Ore": ["Rock", 8], "Khnumium Ore": ["Glowing", 8], "Minosium Ore": ["Glowing", 8], "Morganium Ore": ["Rock", 8], "Persephonite": ["Glowing", 8], "Pyrrhotite Gneiss": ["Hydrocarbon", 8], "Raw Acheronite": ["Hydrocarbon", 8], "Raw Alunite": ["Rock", 8], "Raw Charybdis Voidstone": ["Crystal", 8], "Raw Meteoric Diamond": ["Crystal", 8], "Raw Mica": ["Hydrocarbon", 8], "Raw Scyllan Diamond": ["Crystal", 8], "Raw Tincal": ["Rock", 8], "Raw Voidgem": ["Crystal", 8], "Vanirum Ore": ["Rock", 8], "Yunieon Gas": ["Gas", 8], "Abaddon Ashes": ["Hydrocarbon", 9], "Aesirium Ore": ["Rock", 9], "Asmodeusium Ore": ["Rock", 9], "Astralite": ["Hydrocarbon", 9], "Balderium Ore": ["Rock", 9], "Crude Rutha": ["Hydrocarbon", 9], "Etherion": ["Gas", 9], "Grail Water": ["Hydrocarbon", 9], "Kronosite": ["Glowing", 9], "Modredium Ore": ["Rock", 9], "Nova Dust": ["Crystal", 9], "Noxion": ["Gas", 9], "Pagion": ["Gas", 9], "Raw Erebusite": ["Hydrocarbon", 9], "Raw Promethium": ["Rock", 9], "Raw Star Ore": ["Rock", 9], "Raw Tiberium Crystals": ["Crystal", 9], "Raw Wormstone": ["Crystal", 9], "Star Ash": ["Crystal", 9], "Thothium Ore": ["Hydrocarbon", 9], "Troseki": ["Rock", 9], "Ziosite": ["Hydrocarbon", 9] } """ Error List: Raw Erebusite comes from Hydrocarbon, not Crystal Raw Charon Crystal comes from Glowing, not Crystal? """ """THis list is also defined in functions.php and map_script.js, update all 3! $NAV_TYPE_MAP_BASE = [ "Normal Nav" => 0, "Hidden Nav" => 1, "Normal Gate" => 2, "Faction Gate" => 3, "Normal Station" => 4, "Faction Station" => 5, "Planet" => 6, "Landable Planet" => 7, "Weft" => 8, "Extended Weft" => 9, "Ore Field" => 10 ]; """ def createDB(db_cur): sql = """CREATE TABLE raw_roids( roid_id INTEGER PRIMARY KEY, sector_name TEXT NOT NULL, x INTEGER NOT NULL, y INTEGER NOT NULL, z INTEGER NOT NULL, source_type TEXT CHECK(source_type IN ('Rock','Glowing','Hydrocarbon','Crystal','Gas','Hulk')) NOT NULL, level INTEGER NOT NULL CHECK(level IN (1,2,3,4,5,6,7,8,9)))""" db_cur.execute(sql); sql = """CREATE TABLE raw_roid_contents( raw_roid_id INTEGER NOT NULL, roid_held TEXT NOT NULL, quantity INTEGER NOT NULL, FOREIGN KEY (raw_roid_id) REFERENCES raw_roids(roid_id) ON DELETE CASCADE)""" db_cur.execute(sql); def getNextID(db_cur, table_name, id_col_name): db_cur.execute(f"""SELECT IFNULL(MIN(t1.{id_col_name} + 1),1) AS next_available_id FROM {table_name} t1 LEFT JOIN {table_name} t2 ON t1.{id_col_name} + 1 = t2.{id_col_name} WHERE t2.{id_col_name} IS NULL;""") return db_cur.fetchone()[0] def addDataToDB(db_cur, roid_details, mining_details): #Check if the roid is already in the DB db_cur.execute("SELECT roid_id FROM raw_roids WHERE sector_name = ? AND x = ? AND y = ? AND z = ? AND source_type = ? AND level = ?", roid_details) row = db_cur.fetchone() roid_id = 0 if row: roid_id = row[0] else: roid_id = getNextID(db_cur, "raw_roids", "roid_id") final_roid_details = (roid_id,) + roid_details db_cur.execute('INSERT INTO raw_roids (roid_id, sector_name, x, y, z, source_type, level) VALUES (?,?, ?,?,?, ?,?)', final_roid_details) if mining_details: final_mining_details = (roid_id,) + mining_details db_cur.execute("INSERT INTO raw_roid_contents(raw_roid_id, roid_held, quantity) VALUES (?,?,?)", final_mining_details) #end addDataToDB def parse_log(file_path, channel_name, player_name): results = {} db_conn = sqlite3.connect(":memory:") #db_conn = sqlite3.connect("testing.db") db_conn.row_factory = sqlite3.Row db_cur = db_conn.cursor() createDB(db_cur) sector_pattern = re.compile(r"We have entered (.+?) Sector \((.+?)\)") target_pattern = re.compile(r"\[" + re.escape(channel_name) + "] " + re.escape(player_name) + r": Target '(.+?)' at \(([-\d.]+), ([-\d.]+), ([-\d.]+)\)") target_with_level_pattern = re.compile(r"\[" + re.escape(channel_name) + "] " + re.escape(player_name) + r": ([-\d.]+) Target '(.+?)' at \(([-\d.]+), ([-\d.]+), ([-\d.]+)\)") prospect_pattern = re.compile(r"COMPUTER: Prospected \(([-\d.]+)\) (.+?)$") roid_level_pattern = re.compile(r"\[(\d+)\] " + re.escape(player_name) + r": ([-\d.]+) Target '") current_sector = "" current_roid_level = 0 current_roid = None current_roid_ingame_type = None roid_contents = [] with open(file_path, 'r', encoding='latin-1') as file: for line in file: sector_match = sector_pattern.search(line) if sector_match: sector_name = sector_match.group(1) system_name = sector_match.group(2) current_sector = sector_name current_roid_level = 0 current_roid = None current_roid_ingame_type = None #print(f"We have entered {sector_name} ({system_name})") roid_level_match = roid_level_pattern.search(line) if roid_level_match: current_roid_level = roid_level_match.group(2) hulk_match = target_with_level_pattern.search(line) target_match = target_pattern.search(line) if (hulk_match or target_match) and current_roid is not None: if current_roid_ingame_type != 'Hulk': #Run through once to find max level ore in roid, but not for hulks. for contents in roid_contents: if current_roid[5] < RESOURCE_SOURCE_MAP[contents[0]][1]: current_roid[5] = RESOURCE_SOURCE_MAP[contents[0]][1] #and again to actually update db for contents in roid_contents: addDataToDB(db_cur, tuple(current_roid), contents) roid_contents = [] if hulk_match: target_level = hulk_match.group(1) target_name = hulk_match.group(2) x_coord = int(round(float(hulk_match.group(3)),0)) y_coord = int(round(float(hulk_match.group(4)),0)) z_coord = int(round(float(hulk_match.group(5)),0)) current_roid = [sector_name, x_coord,y_coord,z_coord, target_name, target_level] current_roid_ingame_type = target_name if target_match: target_name = target_match.group(1) x_coord = int(round(float(target_match.group(2)),0)) y_coord = int(round(float(target_match.group(3)),0)) z_coord = int(round(float(target_match.group(4)),0)) current_roid = [sector_name, x_coord,y_coord,z_coord, None, 0] current_roid_ingame_type = target_name prospect_match = prospect_pattern.search(line) if prospect_match and current_roid is not None: quantity = prospect_match.group(1) ore_name = prospect_match.group(2) if ore_name in RESOURCE_SOURCE_MAP: roid_type = RESOURCE_SOURCE_MAP[ore_name][0] roid_type_ingame = "" if roid_type == "Rock" or roid_type == "Glowing": roid_type_ingame = "Asteroid" elif roid_type == "Crystal": roid_type_ingame = "Crystalline Asteroid" elif roid_type == "Gas": roid_type_ingame = "Gas Cloud" elif roid_type == "Hydrocarbon": roid_type_ingame = "Hydrocarbon Deposit" else: roid_type_ingame = roid_type if current_roid_ingame_type != roid_type_ingame: print(f"INVALID SOURCE ERROR: {current_roid} has contents {ore_name} which should come from {roid_type} type roids.") sys.exit(1) current_roid[4] = roid_type roid_contents.append((ore_name, quantity)) db_conn.commit() db_cur.execute("Select * from raw_roids LEFT JOIN raw_roid_contents on roid_id = raw_roid_id") rows = db_cur.fetchall() the_json = dict() prev_roid_id = None for row in rows: # sys.stdout.write("DEBUG: ") # for data in row: # sys.stdout.write(f",{data} ") # sys.stdout.write("\n") if row['sector_name'] not in the_json: the_json[row['sector_name']] = dict() if row['source_type'] not in the_json[row['sector_name']]: the_json[row['sector_name']][row['source_type']] = [] roid_details = { "level": row['level'], "coords": [row['x'],row['y'],row['z']] } if row['roid_held'] is not None: if 'contents' not in roid_details: roid_details['contents'] = dict() roid_details['contents'][row['roid_held']] = row['quantity'] if row['roid_id'] != prev_roid_id: the_json[row['sector_name']][row['source_type']].append(roid_details) else: if row['roid_held'] in the_json[row['sector_name']][row['source_type']][-1]['contents']: the_json[row['sector_name']][row['source_type']][-1]['contents'][row['roid_held']] += row['quantity'] else: the_json[row['sector_name']][row['source_type']][-1]['contents'][row['roid_held']] = row['quantity'] prev_roid_id = row['roid_id'] #print(json.dumps(the_json, indent=4)) print(json.dumps(the_json)) #print(the_json) #print(format_output(the_json)) def format_output(data): formatted_output = "{" for sector in data: formatted_output += f'"{sector}": {{\n' for roid_type in data[sector]: formatted_output += f'\t"{roid_type}": [\n' for inner_data in data[sector][roid_type]: formatted_output += f'\t\t{{\n' for key in inner_data: formatted_output += f'\t\t\t"{key}": {inner_data[key]},\n' formatted_output = formatted_output[:-2] + "\n\t\t},\n" formatted_output = formatted_output[:-2] + "\n\t],\n" formatted_output = formatted_output[:-2] + "\n},\n" # Remove the trailing comma and newline character for the last line formatted_output = formatted_output[:-2] + "}\n" formatted_output = formatted_output.replace("'",'"') return formatted_output if __name__ == "__main__": parser = argparse.ArgumentParser(description="Parse log file") parser.add_argument("file", help="Path to the log file") parser.add_argument("--channel", default="100", help="Channel name (default: [100])") parser.add_argument("--player", default="Doctorje", help="Player name (default: Doctorje)") args = parser.parse_args() parse_log(args.file, args.channel, args.player) "How do I initialize this project?" I assume you are on linux and can get a basic PHP running on your own. sqlite3 better_map.db .read initialization_data/base_schema.sqlite_3 .read initialization_data/3_10_data_import.sql cat initialization_data/mining_data_1 Press "Add Ores", paste in the JSON, submit. cat initialization_data/mining_data_2 Press "Add Ores", paste in the JSON, submit. Edit get_missing_item.php and put in your net-7 username/password. Yes I know that 'isn't secure'. This is a one-person app that I'm likely to be the only user of. It's fine for that. If you do it 'for real', then make it better. Only needed if you want it to auto-fetch missing items (from Hulks) that are not currently in your DB. You should be good to go. The sector_data files are there as backups, in case the 3_10 import fails for some reason, or if you want to debug/test the JSON nav import feature. 4_26_backup_2.7z Edited 6 hours ago by Doctor DB data fix 2 Quote Link to comment Share on other sites More sharing options...
Codemonkeyx Posted March 15 Report Share Posted March 15 (edited) Thanks for providing what you've done so far. It would be a shame for the substantial work you've put into this to go to waste so I might just try and pick this up. :) I'm not much for greenfielding projects but I find I have a lot more enthusiasm for building on ones that already exist. I'll admit I'm not as concerned about what M$ stands to gain commercially from this vs what we would gain from their free hosting, but I don't disagree with your sentiment and will obviously honor your license terms; is this the DoctorV1 License? :P To your point about the chore of data collection I wonder if a lot of it could just be done while playing the game normally with some moderate AHK assist; i.e. instead of a keybind that targets, a hotkey that hits that keybind then prints the position info to a private channel. Over time you would just naturally build up a lot of useful data. The two gaps I see are: 1. There is no sector info in the position print-out which has always annoyed me and I don't know of any way to get that very simple piece of info 2. There is no equivalent to /showdamage which shows the damage you DEAL (and how much was resisted) which people arguably care a lot more about! Seeing the damage types that mobs use is nice, but a way to have the "scrolling combat text" numbers go into the log would make life so much better. Aside from those limitations you could at least get all the position info for things one sector at a time while just playing somewhat normally. If there is no solution for the sector problem that I don't know about there could be an AHK macro to replace gating which copies and clears chat.log everytime you change sectors, then either from context or a pop-up enter a name for the sector, etc. Basically I'm thinking something like the WowheadClient+WowheadLooter that preps and parses as much as it can from chat.log during normal gameplay. Yes, it doesn't satisify the OCD completeness factor (trust me, I get it) but it would focus the data on the things that matter, i.e. the things people using this actually do / care about in-game over time. Edited March 15 by Codemonkeyx grammar, spelling Quote Link to comment Share on other sites More sharing options...
Doctor Posted March 15 Author Report Share Posted March 15 16 minutes ago, Codemonkeyx said: To your point about the chore of data collection I wonder if a lot of it could just be done while playing the game normally with some moderate AHK assist; i.e. instead of a keybind that targets, a hitkey that hits that keybind then prints the position info to a private channel. Over time you would just naturally build up a lot of useful data. The two gaps I see are: 1. There is no sector info in the position print-out which has always annoyed me and I don't know of any way to get that very simple piece of info 2. There is no equivalent to /showdamage which shows the damage you DEAL (and how much was resisted) which people arguably care a lot more about! Seeing the damage types that mobs use is nice, but a way to have the "scrolling combat text" numbers go into the log would make life so much better. Aside from those limitations you could at least get all the position info for things one sector at a time while just playing somewhat normally. If there is no solution for the sector problem that I don't know about there could be an AHK macro to replace gating which copies and clears chat.log everytime you change sectors, then either from context or a pop-up enter a name for the sector, etc. Basically I'm thinking something like the WowheadClient+WowheadLooter that preps and parses as much as it can from chat.log during normal gameplay. Yes, it doesn't satisify the OCD completeness factor (trust me, I get it) but it would focus the data on the things that matter, i.e. the things people using this actually do / care about in-game over time. For Sector data: Look at the script I pasted. It keeps track of what sector you're in via the "Now entering <sector>" line. That's how you keep track of that. For roids: you need to either OCR, image match, or otherwise determine the tech level of the roid. If the emulator wasn't made of lies, then you could just infer that from the highest level ore in the roid (tech 4 roids should always have at least 1 tech 4 ore!). But it becomes a mess of data quickly when you have tech 3 ores dropping and you don't know if they came from legit tech 3 rocks (meaning tech 2 is possible) or from tech 4 rocks (meaning the emulator is a lying sack of shit). So OCR/manual/image match are the only viable strategies. It'd be a simple AHK loop: Push trigger button image match text to get tech level send commands to game client to write manual text like '5' (for a tech 5 roid) to chat send 'ctrl+t' to game client to get it to dump roid position Have your python script keep track of state and associate that ores A and B came out of a tech 5 roid of <type>. This does NOT solve the problem you just finished dealing with on the wiki where both glowing and rock asteroids are called 'asteroid'. Another option is to write a simple C++ voice recognition program which would allow you to simply say "glowing 5" and then mine the rock (assuming the program could write 'glowing 5' to the private channel). I tried doing this, but it's been too many years since I worked with the MS Speech APIs and I don't feel like installing Visual Studio or some other C++ environment on my PC. I tried compiling with cygwin and it was just a PITA. For damage by mob: Correct. This one is always going to be manual, but flying around and poking each mob with 1 shot isn't *that* bad. It gets a bit dicey once you get into the higher level mobs (50+) that really hurt, but maybe someone will run shadowplay or OBS and just record their screen and type it in after the fact. I do agree, some combat logging would be super helpful for automation. If you wanted to go the wowhead path, just write a piece of code that intercepts the packets between the Net7Proxy and the game client. The encryption keys are known, so decrypting the packets is simple, once you can inject yourself into that link. As for what each packet opcode does, find the source cold for the old emulator, the opcodes cannot change as they are hard-coded into the client. That's the 'proper' way to write a wowhead client. But, again, why? The same frustrations that drove me away are the reason I don't feel like writing that. It wouldn't be an accomplishment, it'd be a waste of time. 1 Quote Link to comment Share on other sites More sharing options...
Popular Post Doctor Posted March 25 Author Popular Post Report Share Posted March 25 Since I'm almost out of forum storage, here are some photos showing progress. Posting here gave me the kick in the butt to start working on it some more I guess. https://imgur.com/a/TqFsy9E This is not the final version of what I want to do with ore. Ideally I'd have circular/square/shaped overlays showing where fields are, clearly showing overlapping fields, with mouseovers showing contents in that field. I have the raw data to do all that, but it's considerably more work. And I find it best to collect all the roid data first, and figure out how to define the fields once I have a better idea of all the various edge cases. The raw data is in the database would allow you to mark out where level 5-6 rock-type roids are, for example, but right now, the display doesn't show that. This is not the final form I desire, but rather, just showing what's possible. There is no way to automatically know where a given field is. The only way to figure that out is to mine out very methodically, listen for a field clear bonus, and enter that manually. That would make it possible to have an 'area' calculation wherein you could query all the roids in the sector or within a given X/Y/Z coordinate bounds, then parse through to see if they are part of your desired field, summing up the field drop totals along the way. "Dear god, why?!" Because then you'd be able to see "Oh, almost all of the Titanium comes from these 2 fields" or "Here's where to find Tiberium", etc. Also, the data entry method does not support multiple passes through the same zone. So if I went through Inverness a second time, logging again, it would not average the data correctly; it'd just (roughly) double everything that's already there. Also, if you cherry-pick fields (meaning you only mine some fields and not others) it wouldn't handle that either. Basically, just an FYI that there's a lot of work on those problems needed if someone wants to make a thottbot clone. As for the sector summary, push once to display it, and again to hide it. It only queries the server 1 time to get the data, and keeps it cached until you switch sectors. I will be updating the main post with an updated tarball and parse script shortly. 5 Quote Link to comment Share on other sites More sharing options...
Codemonkeyx Posted March 25 Report Share Posted March 25 That's really cool. It's really interesting seeing all those roid pixels on there and their clustering, etc. Maybe what you need is a drop-down list with checkboxes for each field (just numbered or something) which you can check/uncheck to hide/show, and maybe while that is visible instead of the ore-type color-coding it instead shows a field color-coding so you can quickly see which fields you're interested in and turn off the others in the case of overlaps, then close that and look at the ore-types again to try and figure out whether all the ores you want are in one field or the other or mixed, etc? Seeing this definitely makes me realize all the kinds of things that could be added and how useful this could be! Quote Link to comment Share on other sites More sharing options...
Woodstock HGM [HGM] Posted March 26 Report Share Posted March 26 Nice work, i am following your project with interest . Quote Link to comment Share on other sites More sharing options...
Doctor Posted March 27 Author Report Share Posted March 27 Compare https://www.net-7.org/wiki/index.php?title=Carpenter to https://imgur.com/PBiHmcn That's just beautiful. And it makes it FAR more clear that at Lagrange Point 4: 3 hydrocarbon fields, 1 rock field (probably 2 though) and the hulks are hidden under the other roids, but they are there. Also, comparing the mining summary from the run in the wiki to the one I got just a couple hours ago, it's clear that my initial hypothesis, that certain sectors are better than others for minerals, appears to be false. Carp has a lot of lvl 5 hydrocarbons, so instead of getting mostly Andromesite, I got mostly Hades Blood. In the entire game, there are only 3 possible lvl 5 Hydrocarbon drops (andro, hades, and brood oil). So them swapping around makes sense. It also means Carp appears not to drop brood oil at all. Also, I knew I'd have to make zoom work at some point, but it's looking like I need to make zoom work sooner rather than later. *sigh* I think I'll collect more data first. Collecting data is fun, bashing on javascript and getting the web page to behave is...tedious. It's not as simple as just setting transform and scale on the div. All those little colored squares are in a 2d canvas object, so I need to scale that as well. I think I'll aim to have it work a bit like google earth; scroll wheel to zoom, click-and-drag to move around. Conclusions: I really do have to put some thought and effort into how to average results across multiple clears. *sigh* I was not looking forward to that. I enjoy mining, but that's...probably a bit much. "Let us help!" you'd need to: Create a private channel Press ctrl-t before mining every roid If it's a hulk, manually type in the hulk level before pressing ctrl-t (so, chat would be like: 5 Target 'Hulk' (X, Y, Z) || for a level 5 hulk) I did look into OCRing the screen or using autohotkey ImageMatch. Basically...too tedious. Mine out the entire roid (no cherry picking of any kind) Mine out every field before it has a chance to start respawning Mine out every field in the sector (this can be done over multiple sessions, so long as you do not re-mine a field). Meta: This is a lot for almost anyone to do, plus I'd have to trust the data was gathered correctly...it's...too much. I'll do it myself. Some sectors will be better places than others to find given materials. Not all materials will drop in all sectors, despite the level and roid type indicating they should. Also, Hull Evolution is a major factor in pop rock damage. I have 5 TTs and 1 JE in my mining fleet, block formation. One TT is still on his lvl 100 hull, the others all have their 135 hulls, all have lvl 8 shields, but the lvl 100 hull takes almost 2x as much damage from any given poprock as the other characters. They all have identical gear and are within 100m of the roid. Distance *is* a factor in pop rock damage, but the damage increase I'm seeing happens at all angles (where that lvl 100 hull char is closer/farther based on his group position). So what I am seeing is 100% to do with the hull evolution, and not purely distance. I'll be putting this in the wiki shortly. 2 Quote Link to comment Share on other sites More sharing options...
Codemonkeyx Posted March 27 Report Share Posted March 27 Yeah, for pure fields this is awesome. For mixed fields it's going to be a lot more complicated, especially when they don't have pretty uniform shapes like circles, etc. If you have the concept for each roid of which field it's in (I know that data is challenging as well without fore-knowledge of the fields before you start mining) then you can have the ability to show/hide fields to help sort that out. Either way though, even if there are a few overlapping fields having a good idea of which stuff you can get even if it isn't entirely clear what field clear bonuses you can get is still a huge win. Quote Link to comment Share on other sites More sharing options...
Doctor Posted March 29 Author Report Share Posted March 29 (edited) edit: Also, shameless plug (since some ppl had asked) ^^Makes the account vault click-to-select instead of drag and drop, also adds filters. Huge time saver when you're moving around as much ore and Hulk stuff as I do 😛 I actually managed to mine out the gas fields in Glenn! Someone else was mining the normal roids so I didn't get a full sector mine out, but that's fine, I can get the rest later. But yeah: That's what the gas fields look like. Each of the little 'clusters' you see are one 'field' and represents probably 20 clouds stacked vertically /near each other. I am roughly counting 12 individual fields there. Oh, and the respawn is so fast that as I was clearing the bottom, it had already respawned to the middle hidden nav. o.0 Basically an infinite supply of 8 Gas Boragon 253 8 Gas Yunieon Gas 208 7 Gas Vaneon 202 8 Gas Emperion 190 7 Gas Wexeon 190 Edited March 29 by Doctor Make clusters more obvious, and shameless plug 1 Quote Link to comment Share on other sites More sharing options...
Codemonkeyx Posted March 29 Report Share Posted March 29 The most impressive part of this is that you managed to get the field clears on those gas clouds! 🤣 I think the only ones I've ever been able to get are the ones at the southern edge, and now I can see why as they look a bit smaller than the others. Certainly helpful to know the orientation though, will probably be much easier now! Quote Link to comment Share on other sites More sharing options...
Doctor Posted March 31 Author Report Share Posted March 31 Tarsis needs a redesign. "That doesn't look so bad" "Me brain like rock. Wat problem?" This is a low level zone. OL10s come here. And they leave by OL30. Realistically, the PS is the only character (other than me, and a few other strangers) that will ever mine here. That means characters with 20-22 cargo slots (via: https://www.net-7.org/wiki/index.php?title=Hull_Upgrades#Cargo_Capacity_Comparison) Every single TECH LEVEL of roid here takes up enough cargo slots to reasonably fill a PS's cargo hold, especially when accounting for ammo and the fact that the Red Dragon field guards are out in force, so the PS will need some slots for ammo. Ore fields drop 'sub-level' ores. Meaning a tech 3 roid may have tech 3 and/or tech 2 ores in it. So even if the fields were 1-level wide (only tech 3 roids) you'd still need 15 + 17 = 32 cargo slots to mine out that ONE FIELD. This is insanity. The actual fields, as they are now, are tech 1-3 at the southern end, and tech 3-5 at the northern end. Tech 3-5 is *actually* tech 2-5, so any given field in this sector will take 51 cargo slots to mine. This.is.insane. The fields are mixed, so you cannot focus on one roid type and still get a field clear bonus At OL 30, the station is so far away, and warp is so slow, that you cannot focus on mining out one type of roid, and make it through all 4 types, before the field respawns. Making field clear bonuses (which only exist to ensure explore XP from mining is in-line with other activities) impossible to earn. I'm a OL150 JE with 5 TTs to hold all my mining stuff and damn-near 8000 warp speed. This is infuriating for *me* to mine out. To a newbie PS, this is just soul-crushing. This is not how mining is almost everywhere else in the game. "How 2 fix?" I see these options Separate the fields and make them pure by roid type. Rock fields would need to be changed to be 1-3 and 4-5 though, due to the number of ores in rock-type roids. Other field types would not need level band adjustments. Leave the fields mixed, but disable sub-level spawning (so Tech 3 roids only drop tech 3 ores) and limit each field to a 1-level spread (so ONLY tech 3 roids in a field). And just because, here's a list of all the ores, in their quantities, that dropped before I gave up (this is roughly 90% of a complete mine-out, with minor repeats) Level Roid Source Ore Quantity 5 Crystal Raw Sapphire 7 5 Rock Chromium Ore 5 5 Rock Rhodite 5 5 Glowing Radium Ore 2 5 Rock Titanium Ore 1 4 Glowing Californium Ore 71 4 Glowing Ceresite 36 4 Glowing Saganite 35 4 Rock Silver Ore 26 4 Rock Galactic Ore 20 4 Rock Manganese Ore 20 4 Rock Anubium Ore 16 4 Rock Herculinium Ore 14 4 Crystal Raw Black Opal 14 4 Rock Raw Centauricite 14 4 Crystal Raw Fire Opal 11 4 Crystal Raw Flawless Garnet 9 4 Rock Vanadium Ore 7 4 Crystal Raw Ruby 1 3 Glowing Barite 114 3 Glowing Calcite 104 3 Glowing Alanite 98 3 Glowing Polonium Ore 93 3 Glowing Boronite 78 3 Rock Indium Ore 64 3 Rock Tungsten Ore 61 3 Rock Gallium Ore 60 3 Rock Caesium Ore 55 3 Crystal Raw Malachite 55 3 Crystal Raw Lapis Lazuli 48 3 Rock Star Iron Ore 48 3 Rock Cobalite 42 3 Crystal Raw Citrine 38 3 Crystal Raw Alexandrite 32 2 Glowing Brominite 55 2 Rock Germanium Ore 52 2 Glowing Phosphates 51 2 Glowing Plutonium Ore 50 2 Glowing Cadmium Ore 38 2 Rock Zircon 37 2 Rock Tin Ore 29 2 Rock Magnesium Ore 28 2 Rock Aluminium Ore 26 2 Crystal Raw Tourmaline 22 2 Crystal Raw Turquoise 21 2 Rock Hermesite 18 2 Rock Molybdenum Ore 14 2 Crystal Raw Amethyst 12 2 Crystal Raw Sunstone 8 2 Crystal Raw Moonstone 5 2 Crystal Raw Garnet 4 1 Glowing Sulfates 9 1 Glowing Uranium Ore 9 1 Crystal Raw Onyx 7 1 Crystal Diridium Crystal 5 1 Glowing Lithium Ore 4 1 Rock Lead Ore 3 1 Glowing Potash 3 1 Crystal Raw Agate 2 1 Rock Crude Nickel 1 1 Rock Iron Ore 1 1 Rock Magnetite Ore 1 Quote Link to comment Share on other sites More sharing options...
Doctor Posted April 5 Author Report Share Posted April 5 @Woodstock HGM Since I know you care about R4c this is what the ore fields there look like. I still don't have mouse-over coordinates for navs coded in yet, but maybe someone will find this useful for knowing where to mine in R4c. Mostly lvl 6-7 gas, with some 5 and 8. The non-gas roids are pretty scarce but the rock field has some lvl 9 roids. Quote Link to comment Share on other sites More sharing options...
Woodstock HGM [HGM] Posted April 5 Report Share Posted April 5 nice work Quote Link to comment Share on other sites More sharing options...
bryon3642 Posted April 9 Report Share Posted April 9 Very Cool Indeed Quote Link to comment Share on other sites More sharing options...
Doctor Posted yesterday at 03:47 AM Author Report Share Posted yesterday at 03:47 AM I'm almost done with my first pass through all the sectors, but tonight saw some new stuff I haven't seen before. 1. Did you know Mercury has ring-hulk fields? I've not seen those anywhere else. Note: A decent number of those fields are highly vertical, as in, hulks/roids may be between Z: 15 and Z:-15. I pulled 700 Debris out of Mercury alone. 2. Pluto and Charon has an INSANE amount of ore in it. Most Sectors have around 250 roids in them. P&C? Nearly 600. Ho.lee.schitt. Also, probably one of the best places for Disco in the game. Most of those circles are 30-60km in diameter. I would not be surprised if there are some P&C fields I missed. I have a 22k scan range, but that only does so much. Also, some of the fields are highly vertical, like 15KM above or below the plane where the navs are. 2 Quote Link to comment Share on other sites More sharing options...
Doctor Posted 13 hours ago Author Report Share Posted 13 hours ago And, today, I mined out the last field, in the last sector. Ok, *technically* that's wrong. Paramis, BBW, and Ceres all have uncharted fields, but until i get faction on my TT/TS alts so I don't have to fly back and forth to the station constantly, those are going to remain a mystery. The updated tarball with the current DB of all roids/sectors/navs/etc is attached to the first post. It grew considerably, there's almost 1MB of just roid data... over 15,000 asteroids were mined for this operation. My next steps will involve: Taking a break / playing the game for fun / something else Creating filters so you can find the ores your looking for and know where to look for them. Making the Sector Summary page work (basically a giant 64x194 table of every ore and where it drops in what quantity) I'll drop that on the Wiki once I have it. It won't be perfect, it's one mine-through and 2-3 mine throughs will be needed to get really good data. but, for now, something is better than nothing. Updating the Wiki with screenshots of each sector, at least until better ones exist Seeing if there's a way to save the screenshots as SVG files for better zoom features. Working on adding zoom/pan features to the map. Auto-centering the map based on the navs in the zone, rather than the absolute boundary. If this game had 1000s of players, I know exactly how I could make this into something truly special. But for < 100 people, meh. Still, I learned a lot, and that will be useful going forward. 1 Quote Link to comment Share on other sites More sharing options...
Codemonkeyx Posted 4 hours ago Report Share Posted 4 hours ago Very nice, I had no idea about all that stuff in P&C, that's crazy. That will definitely come in handy! Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.