Skip to content

README.md

Latest commit

 

History

History
31 lines (21 loc) · 720 Bytes

README.md

File metadata and controls

31 lines (21 loc) · 720 Bytes

Ycrawler

Install

git clone https://github.com/assigdev/ycrawler.git

pipenv install

if you don't have pipenv:

pip install pipenv

Run

python crawler.py

Configs

optional arguments:
  -h, --help            show this help message and exit
  -st STATE, --state STATE
                        state file path
  -sl SLEEP_TIME, --sleep_time SLEEP_TIME
                        time for before next parse
  -o OUTPUT, --output OUTPUT
                        output path
  -t TIMEOUT, --timeout TIMEOUT
                        time out for async tasks
  -l LOG, --log LOG     log file path
  -d, --debug           debug logging