Is there a way to create a BIG file quickly?

mike · Joined: 19 Jan 2008 Posts: 29

For a testing application, I need to create big disk files
~1GB quickly.
Creating a small file then using SetEndOfFile makes the big file,
but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
USB1.1 drive).

All I need is for the FAT to be written so the OS (win2k/XP)
thinks it's a valid file. Contents can be garbage.

Is there a way to bypass writing the file contents?
Thanks, mike
--
Return address is VALID!

Archived from group: microsoft>public>vb>winapi

MikeD · Joined: 04 Oct 2007 Posts: 3348

"mike" wrote in message $LN4.5079@trnddc07...
> For a testing application, I need to create big disk files
> ~1GB quickly.
> Creating a small file then using SetEndOfFile makes the big file,
> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
> USB1.1 drive).
>
> All I need is for the FAT to be written so the OS (win2k/XP)
> thinks it's a valid file. Contents can be garbage.
>
> Is there a way to bypass writing the file contents?

Your need/request seems really bizarre to me. Perhaps you could enlighten us
on exactly why you need to do this? Why do you need to create files of 1GB
in size? Mostly, I'm just curious.

There might be a way to quickly create a file of that size, but I'd kinda
doubt it. One way or another, there's got to be that amount of data written
to the file and that's gonna take time. How long depends on many factors,
primary of which is the disk drive itself. You mention USB 1.1. Have you
tried this with USB 2?

You imply (at least how I took it) trying to trick the file system (be it
FAT, FAT32, NTFS, or whatever) into thinking the file is larger than it
actually is. That's just going to cause problems that I don't think you'd
want to have to address.

--
Mike
Microsoft MVP Visual Basic

expvb · Joined: 04 Oct 2007 Posts: 525

Look in MSDN for "Sparse Files".

Ralph · Joined: 04 Oct 2007 Posts: 4148

"mike" wrote in message$LN4.5079@trnddc07...
> For a testing application, I need to create big disk files
> ~1GB quickly.
> Creating a small file then using SetEndOfFile makes the big file,
> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
> USB1.1 drive).
>
> All I need is for the FAT to be written so the OS (win2k/XP)
> thinks it's a valid file. Contents can be garbage.
>
> Is there a way to bypass writing the file contents?
> Thanks, mike
> --
> Return address is VALID!

I too have difficulty why this would be a problem. But when I need a BIG
file for testing I usually just pick a directory and do a "Copy +
+ ..."

-ralph

mike · Joined: 19 Jan 2008 Posts: 29

expvb wrote:
> Look in MSDN for "Sparse Files".
>
>

Windows 2000 NTFS introduces another solution, called a sparse file.
When the sparse file facilities are used, the system does not allocate
hard drive space to a file except in regions where it contains something
other than zeros. The default data value of a sparse file is zero.

The whole objective is to ALLOCATE all the space on the hard drive.
If i understand correctly, sparse files don't do that????
mike

--
Return address is VALID!

mike · Joined: 19 Jan 2008 Posts: 29

MikeD wrote:
> "mike" wrote in message
> $LN4.5079@trnddc07...
>> For a testing application, I need to create big disk files
>> ~1GB quickly.
>> Creating a small file then using SetEndOfFile makes the big file,
>> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
>> USB1.1 drive).
>>
>> All I need is for the FAT to be written so the OS (win2k/XP)
>> thinks it's a valid file. Contents can be garbage.
>>
>> Is there a way to bypass writing the file contents?
>
> Your need/request seems really bizarre to me. Perhaps you could enlighten us
> on exactly why you need to do this? Why do you need to create files of 1GB
> in size? Mostly, I'm just curious.

I'm not at liberty to disclose the why.
The objective is to use up disk space. There's nothing magic about
the 1GB number. It's just a round number and it's smaller than the 2GB
range where other problems happen.
>
> There might be a way to quickly create a file of that size, but I'd kinda
> doubt it. One way or another, there's got to be that amount of data written
> to the file and that's gonna take time.

It seems like one could write the FAT and cause the space to be allocated.
The OS does other stuff that you'd like done in a typical case. But
this ain't a typical case. All I care about is that the next time I
create a file, the OS says, "That there file uses up a gigabyte
and I should allocate the new file elsewhere." I don't intend to ever
read the gigabyte file.
It's just a space filler. It's whole objective is to force the next
file to be created elsewhere on the media. I do want to read/write
other files created around this big space filler.

How long depends on many factors,
> primary of which is the disk drive itself. You mention USB 1.1. Have you
> tried this with USB 2?

Sure, it's a lot faster (but still way too slow) on usb2,
but that doesn't help the usb1.1 requirement.
>
> You imply (at least how I took it) trying to trick the file system (be it
> FAT, FAT32, NTFS, or whatever) into thinking the file is larger than it
> actually is.

NO, I want the OS to be happy that the file is big. I want the OS to
think the space is properly allocated to a valid file. I just don't
care the contents of the file. It can be whatever garbage was on the
media in the allocated space. I'd expect to be able to use the OS to
delete the files when I'm done, but I'd even be ok if I had to reformat
the media.

That's just going to cause problems that I don't think you'd
> want to have to address.
>
This isn't the only way to approach the problem. I just expected it to
be simple. I could accomplish the same thing if I could force a small
file to be created at a particular logical address on the media. I
thought that would require a universal device driver, which would be a
hopeless endeavor for me.

mike

--
Return address is VALID!

mike · Joined: 19 Jan 2008 Posts: 29

Ralph wrote:
> "mike" wrote in message
> $LN4.5079@trnddc07...
>> For a testing application, I need to create big disk files
>> ~1GB quickly.
>> Creating a small file then using SetEndOfFile makes the big file,
>> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
>> USB1.1 drive).
>>
>> All I need is for the FAT to be written so the OS (win2k/XP)
>> thinks it's a valid file. Contents can be garbage.
>>
>> Is there a way to bypass writing the file contents?
>> Thanks, mike
>> --
>> Return address is VALID!
>
> I too have difficulty why this would be a problem. But when I need a BIG
> file for testing I usually just pick a directory and do a "Copy +
> + ..."
>
> -ralph
>
>
Yep, that works. Only problem it that it takes 18minutes/GB and I don't
have 18 minutes (x16) to wait. And even if I weren't impatient, the
laptop battery life ain't that long.
mike

--
Return address is VALID!

SMussler · Joined: 12 Nov 2007 Posts: 21

How about something like this:

Dim fhnd As Long

fhnd = FreeFile
Open "c:\myFile.ext" For Random As fhnd Len = 4096
Put fhnd, 300000, "Hello"
Close 1
MsgBox "done"

That created a file that was 1.14 Gb on my system in 30 seconds.

Steve Mussler

"mike" wrote in message $k15.351@trnddc06...
> Ralph wrote:
>> "mike" wrote in message
>> $LN4.5079@trnddc07...
>>> For a testing application, I need to create big disk files
>>> ~1GB quickly.
>>> Creating a small file then using SetEndOfFile makes the big file,
>>> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
>>> USB1.1 drive).
>>>
>>> All I need is for the FAT to be written so the OS (win2k/XP)
>>> thinks it's a valid file. Contents can be garbage.
>>>
>>> Is there a way to bypass writing the file contents?
>>> Thanks, mike
>>> --
>>> Return address is VALID!
>>
>> I too have difficulty why this would be a problem. But when I need a BIG
>> file for testing I usually just pick a directory and do a "Copy +
>> + ..."
>>
>> -ralph
>>
>>
> Yep, that works. Only problem it that it takes 18minutes/GB and I don't
> have 18 minutes (x16) to wait. And even if I weren't impatient, the
> laptop battery life ain't that long.
> mike
>
> --
> Return address is VALID!

mike · Joined: 19 Jan 2008 Posts: 29

SMussler wrote:
> How about something like this:
>
> Dim fhnd As Long
>
> fhnd = FreeFile
> Open "c:\myFile.ext" For Random As fhnd Len = 4096
> Put fhnd, 300000, "Hello"
> Close 1
> MsgBox "done"
>
> That created a file that was 1.14 Gb on my system in 30 seconds.
>
> Steve Mussler

Thanks for trying.
I cut the file size by 90%. 123MB file took 2minutes 20 seconds
to a usb drive. Bout the same as the other methods I've tried.
I don't have the opion to use a fast computer or a fast drive.
I don't have control of either.
Gotta find a way of stopping the write of the whole file
with zeros and just do the fat.
mike
>
>
>
> "mike" wrote in message
> $k15.351@trnddc06...
>> Ralph wrote:
>>> "mike" wrote in message
>>> $LN4.5079@trnddc07...
>>>> For a testing application, I need to create big disk files
>>>> ~1GB quickly.
>>>> Creating a small file then using SetEndOfFile makes the big file,
>>>> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
>>>> USB1.1 drive).
>>>>
>>>> All I need is for the FAT to be written so the OS (win2k/XP)
>>>> thinks it's a valid file. Contents can be garbage.
>>>>
>>>> Is there a way to bypass writing the file contents?
>>>> Thanks, mike
>>>> --
>>>> Return address is VALID!
>>>
>>> I too have difficulty why this would be a problem. But when I need a BIG
>>> file for testing I usually just pick a directory and do a "Copy +
>>> + ..."
>>>
>>> -ralph
>>>
>>>
>> Yep, that works. Only problem it that it takes 18minutes/GB and I
>> don't have 18 minutes (x16) to wait. And even if I weren't impatient,
>> the laptop battery life ain't that long.
>> mike
>>
>> --
>> Return address is VALID!
>

--
Return address is VALID!

Larry Serflaten · Joined: 04 Oct 2007 Posts: 2644

"mike" wrote

> Gotta find a way of stopping the write of the whole file
> with zeros and just do the fat.

See the tail end of the Remarks section:

SetFileValidData Function
http://msdn2.microsoft.com/en-us/library/aa365544(VS.85).aspx

LFS

Thorsten Albers · Joined: 04 Oct 2007 Posts: 756

mike schrieb im Beitrag
...
> Thanks for trying.
> I cut the file size by 90%. 123MB file took 2minutes 20 seconds
> to a usb drive. Bout the same as the other methods I've tried.
> I don't have the opion to use a fast computer or a fast drive.
> I don't have control of either.
> Gotta find a way of stopping the write of the whole file
> with zeros and just do the fat.

Seems as if you are talking about a USB drive connected to an USB spec. <=
1.1 port. The maximum data transfer rate here is 1,5 MB/second. Creating a
123MB file on an USB drive connected to this port takes at least 123 / 1.5
= 82 seconds = ~1,4 minutes, and creating a 1 GB file at least 1024 / 1.5
= ~683 seconds = 11.38 minutes. But it is not unlikely that additional time
will be consumed since some USB drive controllers work faster, some slower.
And the OS consumes some time too, so that all together it might be very
well that your combination of OS, USB port and USB drive doesn't allow a
file of 123 MB/1GB be created on the USB drive in less then ~2.20/~18
minutes. Note, that a USB 2.0 >>drive>port<<
works 'in' USB <= 1.1, not in USB 2.0. The same is valid for an USB <= 1.1
>>drive>port<<.
On an USB 2.0 drive connected to an USB 2.0 port things would be much
better since USB 2.0 means a maximum data transfer rate of 60 MB/second:
123 MB in 2.05 seconds, and 1 GB in 17.07 seconds.
Note that the data transfer rate additionally depends on other connected
USB devices such as hubs, drives, etc.

--
----------------------------------------------------------------------
THORSTEN ALBERS Universität Freiburg
albers@
uni-freiburg.de
----------------------------------------------------------------------

expvb · Joined: 04 Oct 2007 Posts: 525

It seems from what you have described that you want to fill the disk, so
another process finds the disk or folder "full" or reached a maximum per
folder/disk, and then writes to another folder/disk. Sparse files can be
used for that, but I am not sure if GetDiskFreeSpace() indicates that 1 GB
has been taken. GetFileSize() would return the total file size, including
the empty space, so it returns 1 GB. GetCompressedFileSize() returns the
real size, which is usually smaller.

See "sparse files [Win32]" in MSDN Library Index for the entry point that
describe how to do it. You have to call DeviceIoControl(FSCTL_SET_SPARSE) on
the file before using SetFilePointer/SetEndOfFile to extend the file.

Another option is to is to allocate the sectors themselves without writing
the data. Sysinternals have a program written with VC++ to allocate all the
non allocated sectors, then write zeros to them in a second step. I have
looked at the source, and it can be easily modified to allocate X GB. Here
is a link to the program, but I am not sure if they have the source listed
as before(Search the web for SDeleteSource.zip):

http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx

mike · Joined: 19 Jan 2008 Posts: 29

expvb wrote:
> It seems from what you have described that you want to fill the disk, so
> another process finds the disk or folder "full" or reached a maximum per
> folder/disk, and then writes to another folder/disk. Sparse files can be
> used for that, but I am not sure if GetDiskFreeSpace() indicates that 1 GB
> has been taken. GetFileSize() would return the total file size, including
> the empty space, so it returns 1 GB. GetCompressedFileSize() returns the
> real size, which is usually smaller.
>
> See "sparse files [Win32]" in MSDN Library Index for the entry point that
> describe how to do it. You have to call DeviceIoControl(FSCTL_SET_SPARSE) on
> the file before using SetFilePointer/SetEndOfFile to extend the file.
>
> Another option is to is to allocate the sectors themselves without writing
> the data. Sysinternals have a program written with VC++ to allocate all the
> non allocated sectors, then write zeros to them in a second step. I have
> looked at the source, and it can be easily modified to allocate X GB. Here
> is a link to the program, but I am not sure if they have the source listed
> as before(Search the web for SDeleteSource.zip):
>
> http://technet.microsoft.com/en-us/sysinternals/bb897443.aspx
>
>
Thanks for the input.
I can't find where sdelete allocates sectors before it writes them,
but that's not your fault. Thanks for the tip.
It'll take me a while to get up to speed. My last C program was written
amost 20 years ago in TurboC. I'm unwilling to work on a file deletion
program until I build a system I can trash. Wink

Good news is that I was
able to rebuild the executable...so, guess I have the tools.

Am I having fun yet??? Wink

Thanks again,
mike

--
Return address is VALID!

mike · Joined: 19 Jan 2008 Posts: 29

Larry Serflaten wrote:
> "mike" wrote
>
>> Gotta find a way of stopping the write of the whole file
>> with zeros and just do the fat.
>
> See the tail end of the Remarks section:
>
> SetFileValidData Function
> http://msdn2.microsoft.com/en-us/library/aa365544(VS.85).aspx
>
> LFS
>
>
Thanks, Guys for the input.
SetEndOfFile was one of the first things I tried.

As always, the devil is in the details.

I threw in some breakpoints to take a closer look.
Turns out that all the zero-writing takes place when you close
the file.
So,
Open the file, set the end of file.
Leave the file open,
do other stuff
Move the end of file pointer back to small.
close all the files.
Seems to be doing what I need.

Thanks, again,
mike

--
Return address is VALID!

Karl E. Peterson · Joined: 04 Oct 2007 Posts: 4836

mike wrote:
> For a testing application, I need to create big disk files
> ~1GB quickly.

This take less than 20-40ms here...

Public Sub Main()
Dim stp As CStopWatch
Const tempfile As String = "c:\temp.bin"

Set stp = New CStopWatch
Call MakeBigFile(tempfile, 1024 ^ 3)
Debug.Print stp.Elapsed
'Kill tempfile
End Sub

Public Function MakeBigFile(ByVal FileName As String, ByVal Size As Long) As
Boolean
Dim hFile As Long
Dim mmFile As Long
Const Disposition As Long = CREATE_ALWAYS
Const CreateFlags As Long = GENERIC_WRITE Or GENERIC_READ
Const ShareFlags As Long = FILE_SHARE_READ Or FILE_SHARE_WRITE

If Len(FileName) Then
hFile = CreateFile(FileName, CreateFlags, ShareFlags, ByVal 0&,
Disposition, FILE_FLAG_RANDOM_ACCESS, 0&)
If hFile INVALID_HANDLE_VALUE Then
mmFile = CreateFileMapping(hFile, ByVal 0&, PAGE_READWRITE, 0&, Size,
vbNullString)
If mmFile Then
Call CloseHandle(mmFile)
MakeBigFile = True
End If
Call CloseHandle(hFile)
End If
End If
End Function

> Creating a small file then using SetEndOfFile makes the big file,
> but it gets filled with zeros and takes forever. (~18 minutes/GIG on a
> USB1.1 drive).

The file the above routine creates is filled with zeros.
--
..NET: It's About Trust!
http://vfred.mvps.org

Find .ItemData Quickly With a combobox or listbox, you can send the or messages with SendMessage to quickly find an item in the control's .List array. Is there any similar way to find a value in the ..ItemData array quickly? For example

Quickly Turn on "Show Hidden Files"? Does anyone know the VB statement that will set explorer to show hidden files? I am getting tired of going through the 5 clicks necessary to turn on the global showing of hidden files withing explorer. I would like to write a VB program that does this s

Create AVI file How do i create an .avi file from a series of .tga files??

How to Create Zip File Dear Sir, How to create zip file of any file or folder. Like WindowsXp is create Compress Zip Folder. Sanjay Shah

Create Process that writes to a file in VB6 I had previously been given a solution to this problem but it will only work in .net - I need one that will work in vb6. I am trying to create a process using Createprocess that will write to a file rather than the screen. E.g. Dir >C:\temp.txt The follow