• Same story, different day...........year ie more of the same fiat floods the world
  • There are no markets
  • "Spreading the ideas of freedom loving people on matters regarding high finance, politics, constructionist Constitution, and mental masturbation of all types"

Random IP Browsing Script / Bot with Selenium and Python

Joined
Nov 15, 2016
Messages
331
Likes
149
#1
Hey everybody,

Been feeling motivated lately and decided to share some tools I developed for my self in Python. Web automation has long been my Holy Grail so I can make the machines work for me. Not there yet, day at a time lol. Here's a quick and dirty way to randomize your IP from the command line and a web based anonimizer. This a building block for later applications. Web application testing, account creation, region less web browsing are a few of the utilization of this script.

Full Disclosure, This is all original content owned by me from here

Michigan Python Community
http://www.mipython.com

There are new script tutorials added every day.

Please advise if there's any problems with post or feel free to like page .

Today we are making a script to enter user input into a field on a webpage. Locating the element with Selenium and programmatically sending various key strokes. We are going to navigate from the command line to an instance of Google Chrome. Open up the site kproxy.com . We will take user input from the command line and pass it to submit field. Then click the submit button to surf an IP anonymously.

UPDATE:
This script loads one instance of GOOGLE Chrome then asks for another completely new instance of Chrome for a new url infinatum. Each browser instance has(should, checked and verified so far) its own random IP from kproxy

We should all know how to get here at this point. So, I assume you have Python set up and your libraries installed correctly.

Open up you IDE ( I often just use notepad++)

Lets make a new file called selenium_anon_input.py

First we need to import our modules. Just Selenium for now.
Code:
###### IMPORT SELENIUM
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
Lets define a function called bot1 to do our work for us. This bot is going to just grab an URL for now and open it into a new instance of Google Chrome. As you can see below we first define(def) the function name(bot1) with the Pythonic syntax and indentation.

Ask for user input for the URL to send through the Anonimyzer

Next, we pass the webdriver to variable driver.

Code:
def bot1():
##############    USER INPUT URL IN COMMAND LINE
   url = input("WHAT URL WOULD YOU LIKE TO VISIT ANON? http://")
   ###############   CHROME WEBDRIVER
   driver = webdriver.Chrome()
   ###############   GET URL FOR DRIVER VAR
   driver.get("https://www.kproxy.com/")
Lets run our script and see what happens.
Code:
################ Run Bot1
print(“INITIALIZE BOT 1”)
bot1()
At this point we should be seeing a new instance of Chrome opening up and going to kproxy. Our user input is not doing anything yet. Lets fix that. For this we are going to locate the elements we need on the page with Selenium:

driver.find_element_by_id

To get the elements id name for selenium to use. We simply need to navigate to kproxy with any browser. Right click “inspect element” on the text input field and submit button. Copy and paste this information into the code below.
Code so far.
Code:
###### IMPORT SELENIUM
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

def bot1():
   ##############    USER INPUT URL IN COMMAND LINE
   url = input("WHAT URL WOULD YOU LIKE TO VISIT ANON? http://")
 
   ###############   CHROME WEBDRIVER
   print("DRIVER IS CHROME")
   driver = webdriver.Chrome()

   ###############   GET URL FOR ANONIMYZER
   print("GETTING kproxy.com")
   driver.get("https://www.kproxy.com/")

   ###############    FIND ELEMENT ID MAINTEXTFIELD ####
With the above code we defined a new variable called elem and passed the driver.find_element_by_id method from Selenium to it.

We then created a variable called send_url. That passes the user inputed url as an argument to the Selenium method .send_keys. This passes the url argument to the element id named “maintextfield”.

Now lets click on the submit button and send the url through with the code below.
Code:
############### FIND ELEMENT BY CLASS NAME BOTON
elem = driver.find_element_by_class_name(“boton”)
elem.click()
UPDATE:
Simplified and shortened the find and key submit method in Selenium with this function call:

elem = driver.find_element_by_id(“maintextfield”).send_keys(Keys.ENTER)

Easier and more compact to locate an element and send a key or variable. The original code is commented out around the update.

Our completed script should look like this
EDIT: added a element clear method and instead of clicking on the submit button the RETURN / ENTER key is pressed.
also added an infinite loop that keeps the function bot1() webdriver window open. It’s kinda ghetto but it works for this instance.
You can go back to the terminal and type in a new url after the old one loads. To exit, exit out of terminal or web browser.
Code:
print("*" * 30 )
print("""### MI PYTHON SELENIUM RANDOM IP BROWSER BOT MK2
#  MIpython.com
#  SCRIPT TO USE SELENIUM WEBDRIVER METHODS
#  TO OPEN UP A USER INPUTED URL THROUGH
#  A WEB ANONIMYZER.
     """)
print("*" * 30 )

###### IMPORT SELENIUM
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

def bot1():
   ##############    USER INPUT URL IN COMMAND LINE
   url = input("WHAT URL WOULD YOU LIKE TO VISIT ANON? http://")
 
   ###############   CHROME WEBDRIVER
   print("DRIVER IS CHROME")
   driver = webdriver.Chrome()

   ###############   GET URL FOR ANONIMYZER
   print("GETTING kproxy.com")
   driver.get("https://www.kproxy.com/")

   ###############    FIND ELEMENT ID MAINTEXTFIELD ####
   #elem = driver.find_element_by_id("maintextfield")

   ###############  FIND AND CLEAR FIELD OF ELEMENT NAME MAINTEXTFIELD  ####
       #print("FINDING AND CLEARING ELEMENT")
   elem = driver.find_element_by_id('maintextfield').clear()
    
   ###############     SEND URL VAR TO MAINTEXTFIELD ###
   elem = driver.find_element_by_id("maintextfield").send_keys(url)
   #send_url = elem.send_keys(url)
   #send_url
   ###############  RETURN KEY PRESS
   elem = driver.find_element_by_id("maintextfield").send_keys(Keys.ENTER)
   ###############  FIND ELEMENT BY CLASS NAME BOTON
   #elem = driver.find_element_by_class_name("boton")
   ###############  CLICK ON ELEMENT
   #elem.click()

run = True
while run == True:
   bot1()
else:
   print("press ctrl-c to exit")
####  RUN BOT1 FUNCTION
print("RUN BOT1")
bot1()
 
Last edited:
Joined
Nov 15, 2016
Messages
331
Likes
149
#2
So, I'm writing these tutorials mainly as an educational aid for myself. Also so I don't forget what I'm doing 0.o Thought I'd share in case some others want to get into web automation. This tool is use full for scrapping bing for keywords and saving the result to a local file.

TOPIC MI PYTHON : KW KEY WORD SEARCH BOT WITH PYTHON AND SELENIUM

Hey everybody been having fun working with Selenium in Python . Now lets start putting it all together. Unfortunately some web sites don't want you're bots doing weird stuff to them. So lets incorporate our web scrapper into the Anon redirect function we created earlier. That way at least our real IP shouldn't be banned.

This is a useful tool for SEO optimization.

First lets import everything we need.

Code:
import time
import requests
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
Now instead of using functions individually by themselves. Let's make a class object as a container to organize all our functions into a more coherent data structure.

Below we define our class. Now, apparently a constructor is not called a constructor in python? For ease of my own understanding I'm just going to call the construction of a class a constructor for now. I could be way off with my terminology.

Code:
class KW_SEARCH_BOT(object):
   #   CONSTRUCTOR(?)
   def __init__(self,browser,search_engine_url,kw_list,package):
       self.browser = browser
       #   TEXT OF SCRAPE PAGE   UNUSED  UPON  INITIALIZATION
       self.package = package
       #   CLASS WIDE VAR FOR SEARCH ENGINE TO USE   UPON INITIALIZATION
       self.search_engine_url = search_engine_url
       #   CLASS WIDE VAR FOR KEY WORD LIST   UPON INITIALIZATION
       self.kw_list = kw_list
I called the class KW_SEARCH_BOT. You can call it whatever you want.
1.) We first construct the class with __init__ method.
2.) Then define the class wide variable "browser"
3.) "Package" variable is a blank list upon instantiation not used in this tutorial.
4.) Search engine is the search on we will be using to get our data
5.) Kw_list is our list of keywords.

Now we define the main function for our class.
Code:
def main(self):
  
       ################    REDIRECT THROUGH KPROXY   #####################
       print("OPEN BROWSER")
       self.browser.get("http://www.kproxy.com")
      
       #   FIND ELEMENT
       print("FIND ELEMENT BY ID MAINTEXTFIELD AND SEND SEARCH ENGINE URL")
       elem = self.browser.find_element_by_id("maintextfield").clear()
       elem = self.browser.find_element_by_id("maintextfield").send_keys(self.search_engine_url)
       elem = self.browser.find_element_by_id("maintextfield").send_keys(Keys.ENTER)
       #  WAIT 2 SECONDS
       time.sleep(2)
The above code defines the main function of our class. The object is going to redirect through Kproxy and insert the search engine url of our preferred search engine.
1.) Define the main function self as an object reference to the object.
2.) Get the webdriver.Chrome() as the browser
3.) Find and clear the element "maintextfield"
4.) Print the search engine url in the field.
5.) Hit enter key
6.) Wait 2 seconds

Next we are going to define our main loop to loop through our keyword list.
Code:
###   MAIN LOOP
       #   INJECT KEY WORD LIST
      
       #   LOOPS THROUGH KW_LIST AS KW
       for kw in self.kw_list:
           print("*" * 30)
           print(kw)
           print("*" * 30)
           print("INJECTING KEYWORD PAYLOAD")
           print("-" * 30)
           #   WAIT 3 SECONDS
           time.sleep(3)
           #   LOCATE ELEMENT AND CLEAR
           elem = self.browser.find_element_by_name("q").clear()
           #   LOCATE ELEMENT AND SEND KEYS :  KW VARIABLE
           elem = self.browser.find_element_by_name("q").send_keys(kw)
           #   PRESS ENTER
           elem = self.browser.find_element_by_name("q").send_keys(Keys.ENTER)
           #   WAIT 3 SECONDS
           time.sleep(3)
For the keyword(kw) in the keyword list(kw_list) we send that keyword to the search engine url through the anonimizer.

1.) Make the for loop.
2.) Print the key word currently being posted in the terminal.
3.)Wait 3 seconds.
4.).Locate element "q" and clear the field.
5.) Send the keyword from key word list.
6.) Press enter.
7.) Wait 3 seconds.

Next we are going to print the results of the search into the terminal
Code:
#   PRINT IN TERMINAL
           #   SEARCH RESULTS OF KEY WORD HITS
           hit_count = self.browser.find_element_by_class_name("sb_count")
           #   SEARCH RESULTS OF SNIPPET
           #snippet = self.browser.find_element_by_id("b_results")
           #   SEARCH RESULTS OF B_ALGO
           b_algo = self.browser.find_element_by_class_name("b_algo")
           #   TURN THEWEB OBJECT INTO TEXT AND ENCODE IN UTF-8
           b_algo_text = b_algo.text.encode("utf-8")
           print(b_algo_text)
           print(hit_count.text)
1.) hit_count equals the element with the hit count.
2.) b_algo equals the element with the description of the first item
3.) b_algo_text equals the web elements text encoded in utf-8
4.) Print both hit count and description web objects as text in the terminal.

Now lets save those same results to our local harddrive.
Code:
#   PRINT TO LOCAL FILE
           #   CREATE AND OPEN LOCAL FILE
           local_file = open(  "key_word_results.txt" , "a")
           #   WRITE TO LOCAL FILE KW VARIABLE
           local_file.write(",\n " + kw)
           #   WRITE TO LOCAL FILE HIT COUNT
           local_file.write(",\n " + hit_count.text)
           #   WRITE TO LOCAL FILE DESCRIPTION AS ENCODED STRING
           local_file.write(",\n " + str(b_algo.text.encode("utf-8")))
           local_file.write("\n " + "*" * 30)
           local_file.close()
1.) First we create and open the local file to edit as "key_word_results.txt","a")
2.) Write the current keyword with a line break
3.) Write the hit count with line break.
4.) Write the description as a string of text encoded in utf-8. I had problems with writing chinese characters or other encoding types. I found that this method works for this instance.
5.) Close the file until it loops through again.

Ok, almost done with our first key word scrapping; bot, script, daemon, whatever you feel like calling it. Lets define our key word list as a list variable. Next we need to Instantiate our class.
Code:
#   DEFINE KEW_WORDS LIST TO INPUT
key_words = ["search 1","search 2","search 3"]      
#   INSTANTIATE KW_POST_BOT AS bot
bot = KW_POST_BOT(webdriver.Chrome(),"http://www.bing.com",key_words,[])
Now, the list variable "key_words" has our search words in it. Put the key words you are interested in, in the list separated by a comma.
Instantiate the class object bot with Chrome as the web driver, Search engine url , key_words list , package list unused in this tutorial.

Call the main function of bot object
Code:
bot.main()
Time to run our program and see what happens. You should see a chrome browser window open up and go to the anonimzer url. Then the key words should be injected and the results saved both in terminal and locally.

Here's the complete code.
Code:
print("KW SEARCH BOT")
import time
import requests
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#   DEFINE CLASS INHERIT FROM OBJECT
class KW_POST_BOT(object):
   #   CONSTRUCTOR(?)
   def __init__(self,browser,search_engine_url,kw_list,package):
       self.browser = browser
       #   TEXT OF SCRAPE PAGE   UNUSED  UPON  INITIALIZATION
       self.package = package
       #   CLASS WIDE VAR FOR SEARCH ENGINE TO USE   UPON INITIALIZATION
       self.search_engine_url = search_engine_url
       #   CLASS WIDE VAR FOR KEY WORD LIST   UPON INITIALIZATION
       self.kw_list = kw_list
      
   def main(self):
  
       ################    REDIRECT THROUGH KPROXY   #####################
       print("OPEN BROWSER")
       self.browser.get("http://www.kproxy.com")
      
       #   FIND ELEMENT
       print("FIND ELEMENT BY ID MAINTEXTFIELD AND SEND SEARCH ENGINE URL")
       elem = self.browser.find_element_by_id("maintextfield").clear()
       elem = self.browser.find_element_by_id("maintextfield").send_keys(self.search_engine_url)
       elem = self.browser.find_element_by_id("maintextfield").send_keys(Keys.ENTER)
       #  WAIT 2 SECONDS
       time.sleep(2)
       ###                        MAIN LOOP
       #   INJECT KEY WORD LIST
      
       #   LOOPS THROUGH KW_LIST AS KW
       for kw in self.kw_list:
           print("*" * 30)
           print(kw)
           print("*" * 30)
           print("INJECTING KEYWORD PAYLOAD")
           print("-" * 30)
           #   WAIT 3 SECONDS
           time.sleep(3)
           #   LOCATE ELEMENT AND CLEAR
           elem = self.browser.find_element_by_name("q").clear()
           #   LOCATE ELEMENT AND SEND KEYS :  KW VARIABLE
           elem = self.browser.find_element_by_name("q").send_keys(kw)
           #   PRESS ENTER
           elem = self.browser.find_element_by_name("q").send_keys(Keys.ENTER)
           #   WAIT 3 SECONDS
           time.sleep(3)
          
           #   PRINT IN TERMINAL
           #   SEARCH RESULTS OF KEY WORD HITS
           hit_count = self.browser.find_element_by_class_name("sb_count")
           #   SEARCH RESULTS OF SNIPPET
           #snippet = self.browser.find_element_by_id("b_results")
           #   SEARCH RESULTS OF B_ALGO
           b_algo = self.browser.find_element_by_class_name("b_algo")
           #   TURN THEWEB OBJECT INTO TEXT AND ENCODE IN UTF-8
           b_algo_text = b_algo.text.encode("utf-8")
           print(b_algo_text)
           print(hit_count.text)
          
          
          
           #   PRINT TO LOCAL FILE
           #   CREATE AND OPEN LOCAL FILE
           local_file = open( "date" + "_key_word_results.txt" , "a")
           #   WRITE TO LOCAL FILE KW VARIABLE
           local_file.write(",\n " + kw)
           #   WRITE TO LOCAL FILE HIT COUNT
           local_file.write(",\n " + hit_count.text)
           #   WRITE TO LOCAL FILE DESCRIPTION AS ENCODED STRING
           local_file.write(",\n " + str(b_algo.text.encode("utf-8")))
           local_file.write("\n " + "*" * 30)
           local_file.close()
          
          
#   DEFINE KEW_WORDS LIST TO INPUT
key_words = ["sample1","sample2","sample3"]      
#   INSTANTIATE KW_POST_BOT AS bot
bot = KW_POST_BOT(webdriver.Chrome(),"http://www.bing.com",key_words,[])



bot.main()
 

Goldhedge

Moderator
Site Mgr
Sr Site Supporter
Joined
Mar 28, 2010
Messages
31,503
Likes
39,859
Location
Rocky Mountains
#3
If you could speak English it may help?

You're talking programmer, why would I want to do any of the stuff you are doing above?

To what end? Will it make me smart?
 
Joined
Nov 15, 2016
Messages
331
Likes
149
#4
You're right should be clearer on the function of the script in the description. Basically the web browser is sent to an anonimizer web service before it goes to microsofts bing search engine. Then the script injects the desired key words from the list one at a time while saving that info to the local harddrive. :) Hopefully the SEO stuff is ok too. Trying to drive some bot traffic to a few sites.
 

Goldhedge

Moderator
Site Mgr
Sr Site Supporter
Joined
Mar 28, 2010
Messages
31,503
Likes
39,859
Location
Rocky Mountains
#5
You're right should be clearer on the function of the script in the description. Basically the web browser is sent to an anonimizer web service before it goes to microsofts bing search engine. Then the script injects the desired key words from the list one at a time while saving that info to the local harddrive. :) Hopefully the SEO stuff is ok too. Trying to drive some bot traffic to a few sites.
For what purpose?
 

michael59

heads up-butts down
Platinum Bling
Sr Site Supporter
Joined
Apr 1, 2014
Messages
8,019
Likes
4,495
Location
on the low side of corporate Oregon
#7
Hedge of Gold? Once you are bitten by the code bug its just over. I have flirted with it and just settled on the 'old dog new trick' and dropped it. I think it is kind of cool sending a bot in to raise clickbait. Me? I'm still using crayons. :belly laugh:
 

Goldhedge

Moderator
Site Mgr
Sr Site Supporter
Joined
Mar 28, 2010
Messages
31,503
Likes
39,859
Location
Rocky Mountains
#8
Google rank. Need Christmas monies.
Ahhh, now we're getting somewhere! Why didn't you start with that at the beginning instead of:

"Here's a quick and dirty way to randomize your IP from the command line and a web based anonimizer. This a building block for later applications. Web application testing, account creation, region less web browsing are a few of the utilization of this script."​

Huh?

I suppose a coder could have read your script and figured it out, but as michael59 says 'old dog new trick'...

This is much clearer:

"IF YOU WANT TO IMPROVE YOUR WEBSITE'S GOOGLE RANK... Here's a quick and dirty way to randomize your IP from the command line and a web based anonimizer. This a building block for later applications. Web application testing, account creation, region less web browsing are a few of the utilization of this script."​

Thanks for the clarification....
 
Joined
Nov 15, 2016
Messages
331
Likes
149
#9
Apparently the Internet is more then 50% bots now. Just need to redirect them around with some google foo.