Fetching information from the web.

Discussion of programming on Linux, including shell scripting, perl, python, c/c++, mono, java. Whatever tickles your fancy.

Fetching information from the web.

Postby nahsan » Fri Sep 16, 2005 10:09 pm

Assalam Alaykum,

I need to fetch the forex rates form forexpk.com. The website allows one rate information per page, that means i need to script my way to get em all.
The page has a format like http://www.forexpk.com/kalpoint/kki/cod ... rency=aust
I need to use variable for dd, mm, yyyy, and currency
A big loop to fetch them one by one
I was thinking of writing a script, but then i realized i never wrote one. Anyone can point me to some basic tutorial that can do the job!

Regards,
Nabeel Ahsan
nahsan
Lance Naik
 
Posts: 18
Joined: Tue Jul 01, 2003 8:34 am
ICQ: 6821865
WLM: ahsan_nabeel@yahoo.com
Yahoo Messenger: ahsan_nabeel@yahoo.com
Location: Karachi

Postby Kdaemon » Sat Sep 17, 2005 3:24 am

Use other services like saxobank, forexstreet
I have done a Forex project in lamp, if you need any help then let me know!

what you really want to do with that page content extracting from forexpk ?
reg linux user #298274
Kdaemon
Naib Subedar
 
Posts: 346
Joined: Sat Nov 30, 2002 12:22 pm
Location: Islamabad. GPS: LHR

cURL

Postby ashariqbal » Sat Sep 17, 2005 11:35 am

cURL is probably what you need. http://curl.haxx.se/
ashariqbal
Havaldaar
 
Posts: 105
Joined: Mon Jun 24, 2002 10:01 am
Location: Karachi

Salam

Postby AsadRasheed » Sat Sep 17, 2005 12:59 pm

Dear nahsan,

If you are using linux use wget with "-r" recursive.

Example to mirror whole site is :

$wget -r -m -nv www.killbill.com

if you want to fetch data from particular link than:

$wget -r -m -nv www.killbill.com/Important/

So its download all data in "www.killbill.com/Important/", nothing more .

$cd www.killbill.com/Important/
$tree

and check it.
Dont forget that you should have permission to particular link.

Regards,
M Asad Rasheed
registered linux user #394856
http://www.bsdpakistan.org
AsadRasheed
Battalion Quarter Master Havaldaar
 
Posts: 228
Joined: Fri Jan 28, 2005 6:23 pm
Location: Karachi

Postby nahsan » Sat Sep 17, 2005 8:08 pm

I need the date for the forex rates of the past 5 years, its for forecasting purpose. I need to script my way through using variable so I can change the date , time and currency type. The website allows me to fetch data of a curreny on a particular day.

Regards,
Nabeel Ahsan
nahsan
Lance Naik
 
Posts: 18
Joined: Tue Jul 01, 2003 8:34 am
ICQ: 6821865
WLM: ahsan_nabeel@yahoo.com
Yahoo Messenger: ahsan_nabeel@yahoo.com
Location: Karachi

Postby lambda » Mon Sep 19, 2005 3:02 pm

Code: Select all

import time
import urllib

currency = 'aust'
now = time.time()
start = now - (86400 * 366 * 5)  # a bit over five years ago

while start < now:
  # ltime format is (y, m, d, h, m, s, wday, yday, dst)
  ltime = time.localtime(start)
  url = 'http://www.forexpk.com/kalpoint/kki/code/ver5.1/get.asp?yyyy=%d&mm=%d&dd=%d&currency=%s' % (ltime[0], ltime[1], ltime[2], currency)
  f = file('%s-%y-%02m-%02d' % (currency, ltime[0], ltime[1], ltime[2]), 'w')
  u = urllib.urlopen(url)
  f.writelines(u.readlines())
  u.close()
  f.close()
  print '%y-%02m-%02d done' % (ltime[0], ltime[1], ltime[2])
  start = start + 86400
lambda
Major General
 
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Website: http://www.hungry.com/~fn/
Location: Lahore

Postby atif_majid10 » Tue Oct 04, 2005 2:30 pm

In which programming language you want to write your script. If in PHP, then I can help you in this regard.
Thanks
Atif
atif_majid10
Cadet
 
Posts: 13
Joined: Mon May 31, 2004 3:56 pm
WLM: atif_majid@hotmail.com
Location: Islamabad

Postby lambda » Thu Oct 06, 2005 10:26 pm

writing scripts in php? why would you do that?

i left my last job because of php. never again.
lambda
Major General
 
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Website: http://www.hungry.com/~fn/
Location: Lahore

Re:

Postby LinuxFreaK » Thu Oct 06, 2005 10:34 pm

Dear lambda,
Salam,

lambda wrote:writing scripts in php? why would you do that? i left my last job because of php. never again.


You left naseeb ?

Best Regards.
Farrukh Ahmed
LinuxFreaK
Site Admin
 
Posts: 5132
Joined: Fri May 02, 2003 10:24 am
ICQ: 82075802
Website: http://www.linuxpakistan.net/wiki/index.php?pagename=LinuxFreak
WLM: f4fahmed@hotmail.com
Yahoo Messenger: f4fahmed@yahoo.com
AOL: linuxpakistan@aol.com
Location: Karachi

Re:

Postby lambda » Fri Oct 07, 2005 5:19 pm

LinuxFreaK wrote:You left naseeb ?


yes. that's why they were interviewing people aggressively. the other admin left before i did to work on his phd, and with me leaving, they had no sysadmin.
lambda
Major General
 
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Website: http://www.hungry.com/~fn/
Location: Lahore


Return to “%s” Programming

Who is online

Users browsing this forum: No registered users and 1 guest

cron