Forums

Network is unreachable while running CRON job to store backups into Amazon S3

  1. I have installed s3cmd tool and done configuration settings with 'Amazon s3' Access key and Secret Key and all.
  2. I have created CRON job to make postgresql database backups and store into Amazon S3. However I am getting following error while running CRON job. " "WARNING: Retrying failed request: /db-backups/2013-03-12-backup.tar.gz ([Errno 110] Connection timed out) WARNING: Waiting 3 sec... WARNING: Retrying failed request: /db-backups/2013-03-12-backup.tar.gz ([Errno 101] Network is unreachable)

S3RequestError: Request failed for: /?prefix=db-backups/2013-03-05-backup.tar.gz

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! An unexpected error has occurred. Please report the above lines to: s3tools-bugs@lists.sourceforge.net !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

" "

Please help me on this.

Thanks,

Revanth

I am getting this error while storing DB backups from another server to Amazon s3 using S3cmd tool with CRON job script. Please help me on this.

Is Amazon S3 whitelisted? If not, you won't be able to access it? (Or you will need to get the developers to open it up.)

The whole of amazonaws.com is whitelisted, apparently, so it should be working as long as it's properly configured.

I wouldn't expect a errno 101 in any case, because that indicates that the IP address in question has no valid routing table entry, not that access has been blocked. In more plain terms, the client doesn't even know how to try and connect, as opposed to trying and failing to get through. I don't believe PA blocks access by omitting entries from the routing table, I believe it uses firewall rules instead which would lead me to expect a different error (I may well be incorrect).

I wonder if s3cmd isn't correctly configured, or has failed to read its configuration file properly? Or I guess it's feasibly a DNS error and the hostname is resolving to invalid IP addresses, which would imply either a fault on Amazon's side or possibly PA if there's some sort of DNS relay in effect (I've no idea whether there is or not). However, I think this explanation is the least likely by a large margin.

@revanth, I see from the s3cmd docs that it asks you about using a proxy. Did you set that correctly? The correct settings woudl be proxy.server on port 443 for https and proxy.server on port 80 for http.

@glenn: For reference, am I to infer that the routing table is set up to only allow access to the proxy (and perhaps some other selected systems) on free accounts? I'd assumed that iptables rules or similar would be used. Not to imply either solution is preferable in any way, it's just useful to know the sorts of error message that people might get from particular issues.

Thank you all, for your information. I will follow up as you all suggested.

Thank you all. S3cmd working fine.:)

@revanth -- great! Glad to hear it.

@Cartroo -- we do use iptables; the rule that is getting hit when we block external access looks something like this:

-A OUTPUT -j REJECT --reject-with icmp-port-unreachable

....which I think is why it looks like a routing error.

If you're using icmp-port-unreachable then I'm slightly puzzled, I would expect that to yield errno 111 (connection refused). My understanding was the mapping was:

  • icmp-proto-unreachable = errno 92 (protocol not available)
  • icmp-net-unreachable = errno 101 (network is unreachable)
  • icmp-port-unreachable = errno 111 (connection refused)
  • icmp-host-unreachable = errno 113 (no route to host)

... and the "prohibited" versions exhibited the same behaviour as the corresponding "unreachable" versions, with "icmp-admin-prohibited" also causing an errno 113.

If you're pretty confident that you're using "icmp-port-unreachable" then that'd be interesting to know because it indicates the behaviour must be quite platform-specific.

Hmm, you're right. And 111 is the error we normally get, I just misread the message @revanth was getting.

Perhaps the proxy address was set up incorrectly with an IP on a subnet that really was unreachable? Say, 192.168.0.23?