As with everything, it really depends on your use case and needs.

Areas where NFS excels:

• Read/writes use the standard file system API which is well know
• An app that reads/writes to a local file system can be changed to read/write NFS and without code change be enabled to scale up
• You can scale to an extent, usually by adding hardware/resources to your NFS server
• This is Cloud Foundry specific, but you can get a lot better scale if you have a read-only workload or if you do not need to map uid/gids. When CF mounts your NFS volume, it will use certain mount options to get performance boots if you can work with these restrictions.
• This option tends to work great with legacy apps and getting them onto your cloud platform of choice.

Areas where an HTTP based service or S3-style service excels:

• Scaling. Scales out by adding more nodes/instances to your service, or if you have a service like S3 it is scaled for you and usually to quite large limits (buckets can be multiple TB).
• It depends on the service, but you can usually get better security options (auditing, fine grained access permissions, etc...) this way. NFS is pretty limited with the way it maps uid/gid especially in cloud environments where a uid/gid is not really tied to a specific user.
• It depends on your service, but sometimes you can use the unique properties of a service like S3 to take load off your app, like if you have clients downloading files. S3 or Cloud Front could handle serving those up directly instead of your app spending cycles on it.
• If you're building a greenfield app, this is usually the way to go.

My $0.02. Hope that helps!