{"id":96,"date":"2025-05-02T17:03:25","date_gmt":"2025-05-02T08:03:25","guid":{"rendered":"https:\/\/www.kuruma-ai-life.top\/?p=96"},"modified":"2025-05-02T17:04:28","modified_gmt":"2025-05-02T08:04:28","slug":"%e3%80%90tech-insight%e3%80%91scraping-car-model-images-from-used-car-sites-with-python%e3%80%90development-log-3%e3%80%91","status":"publish","type":"post","link":"https:\/\/www.kuruma-ai-life.top\/?p=96&lang=en","title":{"rendered":"\u3010Tech Insight\u3011Scraping Car Model Images from Used Car Sites with Python\u3010Development Log #3\u3011"},"content":{"rendered":"\n<p>A futuristic scene of three Japanese cars\u2014Prius, Fit, and Note\u2014driving through a smart city, with glowing circuit diagrams and the word \u201cAI\u201d in the background. This symbolizes the launch of our AI-powered car model recognition project.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd0d Why Collect Images from Used Car Sites?<\/h2>\n\n\n\n<p>To train an AI to accurately identify car models, a large and diverse set of real-world images is essential. That\u2019s why I focused on used car listing websites.<\/p>\n\n\n\n<p><strong>Reasons:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Numerous real-world photos are available per car model<\/li>\n\n\n\n<li>Variety in angle, background, and lighting conditions<\/li>\n\n\n\n<li>Easier to treat as labeled data than generic image searches<\/li>\n<\/ul>\n\n\n\n<p>Thanks to this approach, I was able to gather practical, realistic data for training purposes.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\u2699\ufe0f Technologies and Setup<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Language: Python 3.x<\/li>\n\n\n\n<li>Scraping Tools: Selenium + webdriver-manager<\/li>\n\n\n\n<li>Browser Automation: Chrome (headless mode)<\/li>\n\n\n\n<li>Image Downloading: urllib.request<\/li>\n\n\n\n<li>Preprocessing: File size-based filtering of junk images<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83e\udde0 Centralized Car List Management<\/h2>\n\n\n\n<p>I managed car names and search keywords in both Japanese and English like this:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>car_list = [<br>    {\"jp_name\": \"\u30c8\u30e8\u30bf \u30d7\u30ea\u30a6\u30b9\", \"en_name\": \"Toyota Prius\", \"keyword\": \"\u30c8\u30e8\u30bf \u30d7\u30ea\u30a6\u30b9 site:example.com\"},<br>    {\"jp_name\": \"\u30db\u30f3\u30c0 \u30d5\u30a3\u30c3\u30c8\", \"en_name\": \"Honda Fit\", \"keyword\": \"\u30db\u30f3\u30c0 \u30d5\u30a3\u30c3\u30c8 site:example.com\"},<br>    ...<br>]<br><\/code><\/pre>\n\n\n\n<p>\u203b Replace &#8220;example.com&#8221; with the actual domain of the used car site you&#8217;re scraping.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udcf8 Image Collection Workflow<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Access the target used car site<\/li>\n\n\n\n<li>Search for each car model and scroll through results<\/li>\n\n\n\n<li>Extract image URLs<\/li>\n\n\n\n<li>Download images with Python<\/li>\n\n\n\n<li>Filter out small (e.g. banner) images<\/li>\n<\/ol>\n\n\n\n<p>Example:<\/p>\n\n\n\n<p><code>if os.path.getsize(filepath) &lt; 5000:<br>    os.remove(filepath)<br><\/code><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83e\uddfc Filtering Out Unusable Images<\/h2>\n\n\n\n<p>In addition to filtering by file size, I also used a separate script to remove:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Images without cars<\/li>\n\n\n\n<li>Blurry or irrelevant images<\/li>\n<\/ul>\n\n\n\n<p>This ensured the dataset remained clean and useful for training.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udca1 Optimization Highlights<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Adjustable max_images setting<\/li>\n\n\n\n<li>Looped pagination to collect more samples<\/li>\n\n\n\n<li>Random wait times to reduce bot detection risk<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udcc2 What&#8217;s Next?<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Integrate YOLO for automatic car body cropping<\/li>\n\n\n\n<li>Automate dataset split into train\/val\/test for PyTorch<\/li>\n\n\n\n<li>Expand into an app that shows car model + new &amp; used price<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udcdd Final Thoughts<\/h2>\n\n\n\n<p>Used car websites are a powerful source of high-quality training data. With this scraping script, I was able to collect relevant images efficiently and reliably. In the next post, I\u2019ll explain how I trained a ResNet model to classify these car models with high accuracy.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n","protected":false},"excerpt":{"rendered":"<p>A futuristic scene of three Japanese cars\u2014Prius, Fit, and Note\u2014driving through a smart city, with glowing circ<\/p>\n","protected":false},"author":1,"featured_media":49,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[83],"tags":[89,112,91,104,100,90,95,92,103,81,93,109,101],"class_list":["post-96","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-car-recognition-project","tag-ai-development","tag-automotive-ai","tag-car-recognition","tag-computer-vision","tag-deep-learning","tag-image-scraping","tag-machine-learning","tag-python-automation","tag-python-project","tag-resnet-en","tag-selenium","tag-training-data","tag-used-car-dataset"],"jetpack_featured_media_url":"https:\/\/www.kuruma-ai-life.top\/wp-content\/uploads\/2025\/04\/ChatGPT-Image-2025\u5e744\u670822\u65e5-12_44_26.jpeg","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/posts\/96","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=96"}],"version-history":[{"count":2,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/posts\/96\/revisions"}],"predecessor-version":[{"id":98,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/posts\/96\/revisions\/98"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=\/wp\/v2\/media\/49"}],"wp:attachment":[{"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=96"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=96"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kuruma-ai-life.top\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=96"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}