Fetching information from the web.

Discussion of programming on Linux, including shell scripting, perl, python, c/c++, mono, java. Whatever tickles your fancy.
nahsan
Lance Naik
Posts: 18
Joined: Tue Jul 01, 2003 8:34 am
Location: Karachi
Contact:

Fetching information from the web.

Postby nahsan » Fri Sep 16, 2005 10:09 pm

Assalam Alaykum,

I need to fetch the forex rates form forexpk.com. The website allows one rate information per page, that means i need to script my way to get em all.
The page has a format like http://www.forexpk.com/kalpoint/kki/cod ... rency=aust
I need to use variable for dd, mm, yyyy, and currency
A big loop to fetch them one by one
I was thinking of writing a script, but then i realized i never wrote one. Anyone can point me to some basic tutorial that can do the job!

Regards,
Nabeel Ahsan

Kdaemon
Naib Subedar
Posts: 346
Joined: Sat Nov 30, 2002 12:22 pm
Location: Islamabad. GPS: LHR

Postby Kdaemon » Sat Sep 17, 2005 3:24 am

Use other services like saxobank, forexstreet
I have done a Forex project in lamp, if you need any help then let me know!

what you really want to do with that page content extracting from forexpk ?
reg linux user #298274

ashariqbal
Havaldaar
Posts: 105
Joined: Mon Jun 24, 2002 10:01 am
Location: Karachi

cURL

Postby ashariqbal » Sat Sep 17, 2005 11:35 am

cURL is probably what you need. http://curl.haxx.se/

AsadRasheed
Battalion Quarter Master Havaldaar
Posts: 228
Joined: Fri Jan 28, 2005 6:23 pm
Location: Karachi

Salam

Postby AsadRasheed » Sat Sep 17, 2005 12:59 pm

Dear nahsan,

If you are using linux use wget with "-r" recursive.

Example to mirror whole site is :

$wget -r -m -nv www.killbill.com

if you want to fetch data from particular link than:

$wget -r -m -nv www.killbill.com/Important/

So its download all data in "www.killbill.com/Important/", nothing more .

$cd www.killbill.com/Important/
$tree

and check it.
Dont forget that you should have permission to particular link.

Regards,
M Asad Rasheed
registered linux user #394856
http://www.bsdpakistan.org

nahsan
Lance Naik
Posts: 18
Joined: Tue Jul 01, 2003 8:34 am
Location: Karachi
Contact:

Postby nahsan » Sat Sep 17, 2005 8:08 pm

I need the date for the forex rates of the past 5 years, its for forecasting purpose. I need to script my way through using variable so I can change the date , time and currency type. The website allows me to fetch data of a curreny on a particular day.

Regards,
Nabeel Ahsan

lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Mon Sep 19, 2005 3:02 pm

Code: Select all

import time
import urllib

currency = 'aust'
now = time.time()
start = now - (86400 * 366 * 5)  # a bit over five years ago

while start < now:
  # ltime format is (y, m, d, h, m, s, wday, yday, dst)
  ltime = time.localtime(start)
  url = 'http://www.forexpk.com/kalpoint/kki/code/ver5.1/get.asp?yyyy=%d&mm=%d&dd=%d&currency=%s' % (ltime[0], ltime[1], ltime[2], currency)
  f = file('%s-%y-%02m-%02d' % (currency, ltime[0], ltime[1], ltime[2]), 'w')
  u = urllib.urlopen(url)
  f.writelines(u.readlines())
  u.close()
  f.close()
  print '%y-%02m-%02d done' % (ltime[0], ltime[1], ltime[2])
  start = start + 86400

atif_majid10
Cadet
Posts: 13
Joined: Mon May 31, 2004 3:56 pm
Location: Islamabad
Contact:

Postby atif_majid10 » Tue Oct 04, 2005 2:30 pm

In which programming language you want to write your script. If in PHP, then I can help you in this regard.
Thanks
Atif

lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Postby lambda » Thu Oct 06, 2005 10:26 pm

writing scripts in php? why would you do that?

i left my last job because of php. never again.

LinuxFreaK
Site Admin
Posts: 5132
Joined: Fri May 02, 2003 10:24 am
Location: Karachi
Contact:

Re:

Postby LinuxFreaK » Thu Oct 06, 2005 10:34 pm

Dear lambda,
Salam,

lambda wrote:writing scripts in php? why would you do that? i left my last job because of php. never again.


You left naseeb ?

Best Regards.
Farrukh Ahmed

lambda
Major General
Posts: 3452
Joined: Tue May 27, 2003 7:04 pm
Location: Lahore
Contact:

Re:

Postby lambda » Fri Oct 07, 2005 5:19 pm

LinuxFreaK wrote:You left naseeb ?


yes. that's why they were interviewing people aggressively. the other admin left before i did to work on his phd, and with me leaving, they had no sysadmin.


Return to “Programming”

Who is online

Users browsing this forum: No registered users and 1 guest